So, you have inherited a pile of bash without tests
How to test shell scripts using ShellSpec.
I once inherited a pile of bash. There were thousands of lines Perl, Python and of course bash mixed with complicated state and cloud storage.
Everything had been carefully adjusted to be run daily with CRON, and every time daylight saving ended things broke for a day. People were afraid to change anything. It was messy.
I didn’t think a post about bash would become relevant, but hey! Seems bash is more popular than ever.
Throw it all away?
My initial question was “what part of these are actually needed?”. It’s always possible to pull the plug and see who complains. In my case, the scripts were feeding business reporting and payroll calculations.
So, they needed to be rewritten.
To do that, we had to understand them better. What are the inputs and outputs?
That turned out to be a tough question, since the scripts were impure: downloading, transforming and uploading reports to S3. Furthermore, the scripts persisted state on disk and relied on it between runs.
There was also no error handling (no set -e), which meant that some failures that looked unintentional had ended up as “features” along the years.
Tests for … bash?
Writing tests is a good way to get acquainted with bespoke behaviour.
There are established test frameworks for shell scripts. I found a comparison of the main three on this website and a more thorough one in this research repository. I ended up with ShellSpec since I like its shell compatible DSL for writing tests.
With ShellSpec, there’s a couple of different ways to write tests, which may be a source of confusion. Depending on how you want to inject mocks, you can either use command-based mocks or function-based mocks.
For testing existing scripts the command-based mocks are ideal, since no modifications are required for the script under test.
Under the hood, these mocks are extracted into their own script files and prepended to PATH1.
To give you an example, say we want to test the following trivial script:
# my_script.sh - Trivial example
echo "Today is $(date +%Y-%m-%d)"
A ShellSpec test file written using command-based mocks would be as follows:
# spec/my_script_spec.sh - A test file for the trivial example
Describe 'my_script.sh'
Mock date
case "$1" in
'+%Y-%m-%d') echo "2026-01-01" ;;
*) exit 1 ;;
esac
End
It 'prints curent date'
When run script "my_script.sh"
The status should be success
The stderr should be blank
The line 1 of stdout should eq "Today is 2026-01-01"
End
End
There’s a top-level Describe block, and a mock defined with the Mock command.
Test cases look like any other test framework.
With the command-based mocks, there are limits to what can be asserted. We’re limited mainly to file and stdio contents. That’s not a blocker, since I’d rather avoid testing the inner workings2, as the goal was to get rid of the scripts.
A worked example
I mentioned earlier that the actual scripts persisted complicated state on disk and side effects involving S3.
Here’s a more elaborate script, upload.sh to be tested:
#!/usr/bin/env bash
# upload.sh - An example script file representing legacy data pipelines
# ...in reality the scripts didn't use any `set`
set -euo pipefail
# Downloads today's export
download() {
local src="$1"
local dest="$2"
aws s3 cp "$src" "$dest"
echo "downloaded $src to $dest"
}
# Does some complex transform
transform() {
local file="$1"
{
head -n 1 "$file"
tail -n +2 "$file" | awk -F, '$2 != 0' | sort -t, -k2,2nr
} > "${file}.tmp"
mv "${file}.tmp" "$file"
echo "transformed $file"
}
# Uploads report to S3 under a Hive-style date partition
upload() {
local file="$1"
if [[ ! -f "$file" ]]; then
echo "file not found: $file" >&2
return 1
fi
local partition
partition="$(date +yyyy=%Y/mm=%m/dd=%d)"
local dest="s3://amzn-s3-demo-bucket/reports/${partition}"
aws s3 cp "$file" "${dest}/$(basename "$file")"
echo "uploaded $file to $dest"
}
dayiso="$(date +%Y-%m-%d)"
download "s3://amzn-s3-demo-source-bucket/exports/${dayiso}/report.csv" report.csv
transform report.csv
upload report.csv
echo "done"
Here’s a lot more going on. On high-level, it’s an ETL pipeline. It downloads, transforms and uploads some data to S3. I’ve kept it brief here, but in reality, it was more than a thousand lines of bash without functions.
First, since there’s temporary files involved, it’s useful to define a BeforeEach command to isolate the tests.
Describe 'upload.sh'
setup_workdir() { WORKDIR="$(mktemp -d)"; cd "$WORKDIR" || exit; }
cleanup_workdir() { rm -rf "$WORKDIR"; }
BeforeEach 'setup_workdir'
AfterEach 'cleanup_workdir'
# ... test definitions ...
End
Then it’s a matter of mocking out the date and aws CLI commands.
I found simple case statements useful for mock implementations.
Another tip is to log the mock usage to a file, such as aws.calls.
These call logs can then be asserted upon.
The corresponding test file:
# spec/upload_spec.sh - A test file for upload.sh
SCRIPT="$PWD/upload.sh"
Describe 'upload.sh'
# ... test setup ...
Mock date
case "$*" in
'+%Y-%m-%d') echo "2026-01-31" ;; # mock "today"
*) @date "$@" ;; # pass the rest thru
esac
End
Mock aws
echo "aws $*" >> aws.calls # log for test asserts
case "$2" in
cp)
case "$3" in
s3://*/report.csv) printf 'id,total\n1,10\n2,0\n3,30\n' > "$4" ;; # download
s3://*) exit 1 ;; # download
*) [ -f "$3" ] || exit 1 ;; # upload
esac
;;
*) exit 1 ;;
esac
End
It 'downloads report.csv for curent date'
bucket="s3://amzn-s3-demo-source-bucket"
When run script "$SCRIPT"
The status should be success
The stderr should be blank
The line 1 of contents of file "aws.calls" should eq \
"aws s3 cp ${bucket}/exports/2026-01-31/report.csv report.csv"
End
It 'drops zero totals and sorts by total'
When run script "$SCRIPT"
The status should be success
The stderr should be blank
The contents of file "report.csv" should eq \
"$(printf 'id,total\n3,30\n1,10')"
End
It 'uploads report.csv with date partition'
bucket="s3://amzn-s3-demo-bucket"
When run script "$SCRIPT"
The status should be success
The stderr should be blank
The line 2 of contents of file "aws.calls" should eq \
"aws s3 cp report.csv ${bucket}/reports/yyyy=2026/mm=01/dd=31/report.csv"
End
End
The use of date is particularly nasty, as it may be called with dozens of arguments for formatting.
Part of the trick is (unfortunately) to try to standardize the calls in the script, but it’s also handy to define a passthru (“spy”), using the default *) matcher.
# ...
Mock date
case "$*" in
'+%Y-%m-%d') echo "2026-01-31" ;; # mock "today"
*) @date "$@" ;; # pass the rest thru
esac
End
# ...
This way we can freeze the date without implementing all possible call patterns.
With the mock, the following parts in the script work:
# mocked date
dayiso="$(date +%Y-%m-%d)"
# pass thru date command does the formatting
partition="$(date -d "$dayiso" +yyyy=%Y/mm=%m/dd=%d)"
Then what?
After writing some tests and managing to reproduce a happy path (and possibly some failure modes), it’s easier to rewrite the scripts in the preferred language. Since I was working with a large number of scripts, adding the spec as part of CI runs was very helpful.
One thing to point out, though: this case was about stabilizing and replacing old bash scripts. If you’re looking to add tests for new scripts you intend to keep, check out both function-based mocks as well as other frameworks beyond ShellSpec.
The use-case matters when picking a testing framework!