Track which files change during complex operations

Posted by: Paul Whipp 5 years, 6 months ago

When administering systems and deploying code we are often updating large numbers of files. When some of these files change as a result of deployment the implications can be time consuming so we might only want to carry out dependent actions on the server when the file actually changes. Enter my little file tracking utility. Here is the content of file_tracker.py:

"""
Utility to track files and see if they have changed:
checkpoint(file_path) - checkpoints the file
check(file_path) - returns True if the file has not changed since last checkpoint call, False otherwise
"""
import os
import hashlib

__all__ = ['checkpoint', 'same', 'changed']


class NotCheckpointed(Exception):
    pass


def get_info(fname):
    """
    return canonical_fname, fhash
    """
    canonical_fname = os.path.abspath(fname)
    with open(fname, 'rb') as f:
        fhash = hashlib.sha1(f.read()).digest()
    return canonical_fname, fhash

checks = dict()  # canonical filename: sha1 from checkpoint


def checkpoint(fname):
    (canonical_fname, fhash) = get_info(fname)
    checks[canonical_fname] = fhash
    return fhash


def same(fname):
    (canonical_fname, fhash) = get_info(fname)
    try:
        return checks[canonical_fname] == fhash
    except KeyError:
        raise NotCheckpointed


def changed(fname):
    return not same(fname)

With this utility in place, repeated calls to check allow you to verify whether a particular file has been changed or not. Here is how it is called from within a Fabric task where this deploy script only installs new requirements if the requirements.txt file has actually changed:

...
import utils.file_tracker as ft
...
@Fab.task
def deploy():
    """
    Deploy code update to named service and sync/migrate db
    """
    requirements_fname = 'requirements/base.txt'
    site_root = Fab.env.site_setup.site_root

    with Fab.cd(site_root):
        ft.checkpoint(requirements_fname)
        run('find -name "*.pyc" -exec rm {} \;')
        run('git pull')

        if ft.changed(requirements_fname):
           update_requirements()

    migrate()
    collectstatic()

Comments

There are currently no comments

New Comment

required

required (not published)

optional

Australia: 07 3103 2894

International: +61 410 545 357