07. sos Report Data Pack Layout Written on . Posted in sos command.
When an sos report is generated, it outputs an encrypted and compressed tar file. Inside this file lies a well-organized structure, reflecting the myriad Linux sub-systems from which the data is collected.
The contents of an sos-report allow engineers to view a snapshot of the system across multiple layers of complexity.
When a sos-report file is unpacked, it creates a directory structure like this:
. ├── ./boot ├── ./etc ├── ./lib -> usr/lib ├── ./opt ├── ./proc ├── ./run ├── ./sos_commands ├── ./sos_logs ├── ./sos_reports ├── ./sos_strings ├── ./sys ├── ./usr ├── ./var └── ./EXTRAS 13 directories
All these directories with the exception of sos_commands, sos_logs, sos_reports, sos_strings and EXTRAS are partial copies of a regular Linux system and the contents of these directories will be nostly configurations files.
- The EXTRAS directory is not part of a normal sos-report filesystem however sos-vault creates this directory when the sos_extras plugin was used to include custom files in the sos report data.
- The sos_strings directory contains a 30 days dump of the Linux journal an can be quite large.
- The sos_reports directory contains a full listing of the files and commands executed that are included in this sos-report in text format, json format and html format:
- The sos.json is the index of the sos-report in json format. It is used to find any file or command output in the sos directory structure. Each json object listed in this file has an "href" field that specifies the file location in the sos-report directory structure.
- The manifest.json file in the sos_reports directory describes how the execution of the sos report went, it includes execution timestamps, commands executed with parameters, files copied for every plugin and obfuscation process data done during the sos report activity.
- The sos_logs directory contains the log of the sos execution and any error found during it execution can be seen in the sos.log file.
- The sos_commands directory contains a large set of subdirectories each one corresponding to each of the different sos plugins that were executed during the sos-report task. Inside each plugin-subdirectory are store the output of the commands related to that specific plugin. The sos_commands directcory structure looks like this:
. ├── ./sos_commands │ ├── ./sos_commands/apparmor │ ├── ./sos_commands/apport │ ├── ./sos_commands/apt │ ├── ./sos_commands/ata │ ├── ./sos_commands/block │ ├── ./sos_commands/boot │ ├── ./sos_commands/canonical_livepatch │ ├── ./sos_commands/cgroups │ ├── ./sos_commands/console │ ├── ./sos_commands/cron │ ├── ./sos_commands/cups │ ├── ./sos_commands/date │ ├── ./sos_commands/dbus │ ├── ./sos_commands/devicemapper │ ├── ./sos_commands/devices │ ├── ./sos_commands/docker │ ├── ./sos_commands/dpkg │ ├── ./sos_commands/filesys │ ├── ./sos_commands/firewall_tables │ ├── ./sos_commands/fwupd │ ├── ./sos_commands/gdm │ ├── ./sos_commands/grub2 │ ├── ./sos_commands/hardware │ ├── ./sos_commands/host │ ├── ./sos_commands/i18n │ ├── ./sos_commands/kernel │ ├── ./sos_commands/libraries │ ├── ./sos_commands/libvirt │ ├── ./sos_commands/login │ ├── ./sos_commands/logrotate │ ├── ./sos_commands/logs │ ├── ./sos_commands/lvm2 │ ├── ./sos_commands/memory │ ├── ./sos_commands/mysql │ ├── ./sos_commands/networking │ ├── ./sos_commands/networkmanager │ ├── ./sos_commands/nodejs │ ├── ./sos_commands/pam │ ├── ./sos_commands/pci │ ├── ./sos_commands/perl │ ├── ./sos_commands/process │ ├── ./sos_commands/processor │ ├── ./sos_commands/python │ ├── ./sos_commands/release │ ├── ./sos_commands/scsi │ ├── ./sos_commands/services │ ├── ./sos_commands/snap │ ├── ./sos_commands/soundcard │ ├── ./sos_commands/ssh │ ├── ./sos_commands/sunrpc │ ├── ./sos_commands/systemd │ ├── ./sos_commands/sysvipc │ ├── ./sos_commands/ubuntu │ ├── ./sos_commands/udisks │ ├── ./sos_commands/ufw │ ├── ./sos_commands/usb │ ├── ./sos_commands/wireless │ └── ./sos_commands/x11 ├── ./sos_logs ├── ./sos_reports └── ./sos_strings ├── ./sos_strings/logs └── ./sos_strings/networkmanager
For example, inside the process plugin-directory (process information) the following files can be found:
./sos_commands/process ├── ./sos_commands/process/lsof_M_-n_-l_-c ├── ./sos_commands/process/pidstat_-p_ALL_-rudvwsRU_--human_-h ├── ./sos_commands/process/pidstat_-tl ├── ./sos_commands/process/ps_alxwww ├── ./sos_commands/process/ps_auxwwwm ├── ./sos_commands/process/ps_axo_flags_state_uid_pid_ppid_pgid_sid_cls_pri_addr_sz_wchan_20_lstart_tty_time_cmd ├── ./sos_commands/process/ps_axo_pid_ppid_user_group_lwp_nlwp_start_time_comm_cgroup ├── ./sos_commands/process/ps_-elfL └── ./sos_commands/process/pstree_-lp 0 directories, 9 files
As you can see, the names of the files correspond to the Linux command that was executed along with all the options used. In this example the lsof command was executes with the -M -n -l and -c options and the output of the command is the contents of the file as shown in the next screenshot:
This is the same behaviour for every plugin included during the sos report execution.
Exploring a sosreport is challenging due to the sheer volume of logs, configurations files, and system command outputs makes manual analysis time-consuming and inefficient. Security concerns and the need for log correlation further complicate the process.
To address these challenges, tools like sos-vault can revolutionize sosreport analysis by structuring data, improving searchability, and leveraging AI for automated issue detection. By streamlining troubleshooting, such solutions can significantly reduce the time and effort required to diagnose Linux system issues.