Akom's Tech Ruminations

Various tech outbursts - code and solutions to practical problems

Docker: Automatically remove containers that have been running too long Linux

Posted by Admin • Thursday, October 20. 2016 • Category: DevOps, Linux

Why Because my Jenkins setup sometimes starts containers and forgets about them. Either it thinks it failed to start one, or the container itself has trouble starting. Either way, I'm left with containers that are running, trying to connect to Jenkins in vain, forever. The proper way to fix this is probably to have the containers timeout at some point, but that mechanism is broken.

Anyway, the fix I have is a true hack: find containers that have been up more than 2 days and kill them. None of our jobs should run for more than about a day, so this is a safe limit. Here is a bash script to do this:

The only hard part here is converting a human-readable age from docker output (32 hours) to a number we can use for comparison. Fortunately, passing "32 hours" to "date -d" gives you a time 32 hours in the future, so from there it's easy.

#!/bin/bash

DOCKER_CMD="docker -H tcp://localhost:YOUR_PORT"
# max seconds before containers should be killed
KILLTHRESHOLD=${1:-172000} #default max age in seconds

if [ "$KILLTHRESHOLD" -gt 0 ] ; then

    $DOCKER_CMD ps --format "{{.ID}}:{{.RunningFor}}" | while read line ; do
        ID=$(echo $line | cut -d : -f 1)
        HUMANAGE="$(echo $line | cut -d : -f 2)"
        FAKETIMESTAMP=$(date -d "$HUMANAGE" +%s)
        AGE=$(echo "$FAKETIMESTAMP - $(date +%s)" | bc)
#        echo "Age is $AGE for $ID"
        if [ "$AGE" -gt "$KILLTHRESHOLD" ] ; then
            logger -t dockerwatch "Killing $ID, running for $HUMANAGE"
            $DOCKER_CMD stop "$ID"
            sleep 5
            $DOCKER_CMD kill "$ID"
            $DOCKER_CMD rm   "$ID"
        fi
    done

fi
 

0 Trackbacks

  1. No Trackbacks

0 Comments

Display comments as (Linear | Threaded)
  1. No comments

Add Comment


You can use [geshi lang=lang_name [,ln={y|n}]][/geshi] tags to embed source code snippets.
Enclosing asterisks marks text as bold (*word*), underscore are made via _word_.
Standard emoticons like :-) and ;-) are converted to images.

To prevent automated Bots from commentspamming, please enter the string you see in the image below in the appropriate input box. Your comment will only be submitted if the strings match. Please ensure that your browser supports and accepts cookies, or your comment cannot be verified correctly.
CAPTCHA

What is the primary language of this blog? (Anti-SPAM question)


Submitted comments will be subject to moderation before being displayed.