Simple system for recording all bash commands on a laptop or network, without security

There are three components:

• bash.logging, a bash script to initially set up the logging tool in a login session
• syslog, set up to receive a message every time there is a new bash prompt
• /var/log/commandline.log, containing all lines typed at all bash prompts

Here’s how it works:

Many networks allow their users shell access on Unix machines, and administrators always have access. The shell is often bash (the Bourne-Again Shell) which offers many conveniences. bash also presents the users and network administrators with problems. Some of these problems can be addressed with effective auditing, that is, recording every command along with context such as time, date, machine, user and process id. Such a system can be used standalone or integrated into a more general centralised logging system. Several solutions to this are available online, but as of April 2008 none of them were very good.

Motivations for Cloba

At some point everyone who uses a commandline wants to know “What was typed?” after the fact. Often users really want to know more than that, such as “What was typed, by whom, in what process and on what network host and when?” Reasons include:

• Documentation: Writing down how to achieve a particular clever result found after much trial and error
• Reconstructing events: Not all commandline users are experts! If all commands are logged it is possible to work out how a user got themselves or other people into trouble… “but I never type rm -rf!”
• Big-brother trend analysis: what facilities are regularly used on our network? Are there some common mistakes?
• Big-brother psychological security: users are the biggest problem on networks, often because they don’t know things they should, or forget these things. Knowledge that all commands are logged may motivation for thinking before acting
• Auditing root access: tools such as sudo help, but what admin doesn’t regularly use sudo su - to solve problems where sudo gets in the way?
• After-the-fact forensics: a server has crashed mysteriously… did someone type things that may have caused that to happen?
• After-the-fact security auditing: there’s been a breakin… chances are the intruder wasn’t able to cover his tracks perfectly on all machines. Now you can look.

Problems We Aren’t Solving!

• How to do this securely, against a Cloba-specific attack. Circumventing Cloba can take just a few seconds, although without extra effort the action of circumvention may be recorded.
• Provision of a specific audit trail. Cloba provides a general audit trail, in the sense that across many instances and many sessions on many machines patterns become obvious that are very difficult to fake. However, as the previous point makes clear, injecting false data or withholding data from Cloba is not hard.
• What to do with the history data once it is collected. For example, providing a sort of inverse mechanism of a centralised infinite history recall.

Bash History and its Problems

Bash has an unreliable history mechanism: bash commands are written to a file, but this file cannot be trusted to be complete, and there is no way of predicting which of multiple bash sessions wrote to it. The trivial case solved by the history file is where what you are looking for happens to be in the last 500 commands (the default) and in the history file known to the current shell instance. Apart from this case, there is no auditing facility for bash.

Problems with bash are therefore:

• loss of work: users often re-solve the same problems at commandline time after time
• lack of auditing: it can be very helpful to know exactly what was typed at a known point in the past, for example to diagnose a problem
• single-user: bash is inherently single-user. In a multi-user environment it can be enough to know who was typing commands around a particular time, which can narrow a problem down enough to solve with a quick phonecall.
• traceability: the bash history is an important part of computer forensics and security pattern matching, despite being fakeable and destructible. Anything that improves on the bash history functionality and reliability contributes to a better environment for after-the-fact security analysis, even if it is not useful as a front-line security tool.

Somewhat Related Solutions

There are many ways of imposing control. Two of these that focus on logging are:

• BASHA ↗, a modified bash that logs all commands.
• Snare ↗, a modified Linux kernel for logging to meet C2 standards.

This class of solutions is not suitable for soft-touch retrofitting, because people will have to change their work habits and that means engagement with the userbase. In contrast, Cloba is about improving logging in the network you already have.

Goals for Cloba

• Record lines every time user presses return at bash prompt
• Approximately ACID-like in effect: if data has been written by Cloba then it will be internally consistent, there will be no half-written transactions, and data cannot be overwritten
• Be portable to any unmodified bash on any platform
• Easily adaptable to centralised network logging as well as for just one machine
• Simple. Any administrator should be able to understand what is going on
• Available to end-users. At the administrator’s discretion, it will be possible to let anyone with with credentials for an account to see and search the entire history for that account.

The Primitives

There are quite a few potential tools. The solution we converged to used five of them, but we could have easily chosen other solutions:

• fc - a bash built-in command for editing the commandline
• logger - a shell command interface to the syslog(3) system log module
• declare -r - the way to make a bash variable read-only
• trap DEBUG - trap is a signal (interrupt) handler, and the special signal DEBUG tells bash to execute the supplied handler after every command
• PROMPT_COMMAND - a variable that bash executes as a command before displaying the primary bash prompt

A Common Bad Solution

Web searches give variations on this:

trap 'logger $LOGNAME: "$(fc -ln -0)"' DEBUG

which has these problems:

• logs the wrong command (try it!)
• logs too many commands (every line in a for loop…)
• can be disabled very easily, even accidentally

Developing the Solution

Scenario 1: The Single User

If a user, for example with a laptop, wishes to log all commands to a file for future reference, the trivial answer is:

export PROMPT_COMMAND='history 1 >> $HOME/logfile.$$'

The shell executes the command in single quotes just before reading a new command. You can try this at the command prompt within a single session, or put it at the end of your .bashrc file to be executed every time you log in.

Pros:

• simple. Try this one liner today!
• more reliable than saving the history when the shell exits. If you want you can arrange for this file to end up in the bash history file, although you’ll need to think about managing how big you want it to be and what pruning criteria you use.
• different sessions use different files (the $$ adds the process ID to the name.) Alternatively, move the “.$$” to the line being appended commands will be logged to the same file (the underlying operating system makes sure lines are not overwritten or missed.) Lines from different shells will be interleaved.

Cons:

• a command line ending in ‘; exit’ doesn’t get logged, because the shell never asks for the next command (and if BASH_HISTORY is set to /dev/null the command will not appear in any file)
• the command executes just before the user types the first command; if there is anything in the .bash_history file, it gets logged at that point as if it had been typed as first command of the new session
• an empty command line results in the last command being logged a second time; if a lot of commands are typed, and some repeated, this gets confusing
• if the user has another use for PROMPT_COMMAND this will interfere
• easy to disable, even by accident

Scenario 2: A Multiuser Machine

Assuming every user has this command active (see below for how to do this on specific operating systems) a multiuser machine can keep consistent logs quite reliably:

# Command logging

__LAST_COMMAND="$(history 1)"

declare -r BASH_COMMAND

declare -r __TRAP='
__THIS_COMMAND="$(history 1)"
if [ "$__LAST_COMMAND" != "$__THIS_COMMAND" ]
then
    __LAST_COMMAND="$__THIS_COMMAND"
    if [ "$PROMPT_COMMAND" != "$BASH_COMMAND" ]
    then
    '"$(which logger)"' -p local7.notice -- "-- CMD -- $LOGNAME@$HOSTNAME[$$] ($PWD): $__LAST_COMMAND"
    fi
fi'

declare -r PROMPT_COMMAND='trap "$__TRAP" DEBUG'

declare -r HISTCONTROL=''

logger -p local7.notice -- "-- CMD -- $LOGNAME@$HOSTNAME[$$]: logged in"

This solution uses logger to send a syslog message for every line typed. logger is good because:

• writes to secure area without any privilege required (usually in /var/log/ by default)
• handles concurrent sessions from concurrent users without any contention

Pros:

• Can’t be so easily disabled (notice -r for readonly in declare -r). You can’t disable this without using some other shell, or compiling your own. The fact that typing tsch on most systems will mean commands are not logged is not a fatal problem according to the goals listed earlier.
• No repeat entries caused by pressing return on a blank line. That is the purpose of the __LAST_COMMAND logic.

Cons:

• if you have ignoredups or erasedups in your $HISTCONTROL duplicate commands will not be logged. If you really want to log everything you should set HISTCONTROL=’’’’'
• all logging goes into the default syslog destination, user.notice by default, which usually ends up in /var/log/messages. This is addressed in the next section.

if [ -r /etc/bash.logging ]
then
   . /etc/bash.logging
fi

Scenario 3: An Entire Network

Logging commands centrally is simply a matter of changing the syslog configuration, something network administrators should consider doing anyway:

• logger -p localX.levelY sends log messages to the service localX at priority level Y, eg logger -p local3.info. Using this you can configure a specific destination in syslog.conf, such as /var/log/cmdline.log.
• if a machine’s syslog is already setup to send to a remote destination, there is nothing more to do
• an example catchall syslog rule to put at the end of syslog.conf is *.* @mycentraldest.example.org

When you are logging all commands typed everywhere on a network, each line needs more information. This is a replacement for the logger line in the previous script:

'"$(which logger)"' -p local3.notice -- "-- CMD -- $LOGNAME '"$(pwd)"'@$HOSTNAME[$$]: $__LAST_COMMAND"

which will give log results like this for the command ls /dev/null:

<syslog default prefix> -- CMD -- /data/example/ myusername@bigserver[22187]:    35  ls /dev/null

With the usual syslog data to the left of the – CMD flag in the log file.

syslog records the username, but the $LOGNAME is still needed because syslog does not change the name to reflect any su commands.

What Doesn’t Get Logged

For version of bash prior to 3.25 (and maybe earlier, but have not done testing!) the DEBUG trap behaviour has been brought in line with documentation, which says “DEBUG only fires when a line will be executed.” For these later versions of bash, any lines that bash can’t execute don’t get logged although they are added to the bash history. These include:

• comment lines (starting with #)
• invalid bash commands (for example, this phrase including its enclosing parentheses)

For example, if you’re using Debian etch, with bash 2.0.5, these lines will be logged. If you’re using Ubuntu Gutsy Server, with bash 3.25, they will not be.

Download It!

Putting everything together… here is a download that gives you the full solution for the multiuser case: bash.logging