Use git to sync dotfiles

It is a common practice to use `git` to back-up and sync ones "dotfiles". I also create Python script to help automate this process.
Contents hide
1 Rationale
2 Setup
3 A Python automation script
3.1 The script
3.2 Breakdown of the script
3.3 The workflow
3.3.1 Initial setup
3.3.2 Make changes to the dotfiles and sync the changes
3.3.3 Get the dotfiles on a new system installation

Rationale

For those Linux or Mac users, dotfiles are text files containing various configurations of the operating system or softwares wherein. For instance, .bashrc is the config file for the bash shell, .vimrc for the vim text editor, .gitconfig for git etc.. They typically have a file name starting with a single dot, which denotes hidden files in UNIX-like systems. Thus comes the nickname dotfiles.

Thanks to the great customizability of the Linux system, it is not uncommon for Linux users to spend quite some time on tweaking and customizing their system and softwares. It would be a great loss of effort if one has to repeat such customization processes every time he re-installs the system, migrates to a different Linux distro or replicates the same setup on a second machine. The good news is that such dotfiles are plain text files, which can be easily backed-up or synced using a git repository. If you browse through the GitHub pages, you will find many git repos containing someone’s dotfiles. This is what we’ll be talking about in this post.

Setup

I’m working on a Linux machine. Procedures in Mac will be pretty much exactly the same. Windows doesn’t have dotfiles, but you can sync some other configuration files just like any other normal text files via git.

The dotfiles for a specific user are typically stored in 2 different locations inside the user’s HOME directory:

  1. in the HOME folder itself, with a file name starting with a dot so they are "hidden" in a sense that they won’t be shown in the ls shell command, or your graphical file manager window. To see these hidden files, you will have to use the ls -a shell command, or toggle the show hidden files option in the GUI file manager.
  2. in a .config folder inside the HOME folder. Note that the folder name also has a leading dot so it is hidden as well.

There aren’t any strict rules to decide in which of these 2 locations a specific config file will be located, it is pretty much up to the software or system developers. You would normally find system-related config files in the HOME folder itself (location 1), like .bashrc, .profile, and .ssh; and software-related config files in the .config folder (location 2), like ~/.config/chromium for the chromium browser, ~/.config/inkscape for the inkscape image-editor and ~/.config/rofi for the rofi launcher. But there are many exceptions, and in many cases you can customize the desired location through some tweaks.

The basic idea is to create a dedicated folder somewhere in the file system to house the dotfiles, and make a git repo out of the folder (for basic usage of git, checkout this article.) Then move some selected dotfiles/folders into the git repo, before creating a symlink for each moved file/folder back from the git repo to their original locations so they stay functional. With the dotfiles stored in a git repo, they can be synced up with a remote service (whether it being GitHub, bitbucket or GitLab). Next time we do a fresh system install these dotfiles can be pulled down from the remote and accomplish the customization easily.

A Python automation script

The above procedures should be fairly straightforward if you are already familiar with the git workflow. However, there is nothing that prevents us from further streamlining the process by creating a script to automate it. Below is a Python script I created to help myself backing-up, copying into a git repo and installing to a new machine from git.

The script

Paste the code first:

'''Copy selected dot files/folders to a backup location and a git
folder for sync, and create symlinks from git folder back to original
places under HOME folder.
'''
import os,sys
import shutil
import argparse

GIT_DIR="~/.dotfiles"           # git repo dir
BKDIR="~/.dotfiles_bk"          # backup dir

# list of files/folders to sync and backup
FILES=[
'.profile',
'.bashrc',
'.gitconfig',
'.vimrc',
'.vim',
'.pdbrc',
'.tmux.conf',
'.tmux',
'.config/ranger',
'.config/rofi',
'.emacs.d/snippets',
'.emacs.d/init.el',
'.emacs.d/config.org',
'.emacs.d/templates',
'.emacs.d/org-protocol.desktop',
'.emacs.d/.org-id-locations',
'.emacs.d/create-word-drill.sh',
]

def copy2(src,target,overwrite=True):
    '''use copytree() for folders, copy2() for files'''
    if os.path.isdir(src):
        if os.path.exists(target):
            if overwrite:
                shutil.rmtree(target)
                shutil.copytree(src,target)
                print('\n# <copy2>: Copy and overwrite', src, 'to target', target)
        else:
            shutil.copytree(src,target)
            print('\n# <copy2>: Copy', src, 'to target', target)
    else:
        if os.path.exists(target):
            if overwrite:
                os.remove(target)
                shutil.copy2(src,target)
                print('\n# <copy2>: Copy and overwrite', src, 'to target', target)
        else:
            shutil.copy2(src,target)
            print('\n# <copy2>: Copy', src, 'to target', target)
    return

def move2(src,target,overwrite=True):
    if os.path.exists(target):
        if overwrite:
            try:
                shutil.rmtree(target)
            except:
                os.remove(target)
            shutil.move(src,target)
            print('\n# <move2>: Move and overwrite', src, 'to target', target)
    else:
        shutil.move(src,target)
        print('\n# <move2>: Move', src, 'to target', target)
    return

def _doBackup(folder):

    #----------------Loop through files----------------
    for ff in FILES:
        abpathff=os.path.join('~',ff)
        abpathff=os.path.expanduser(abpathff)
        bk_path=os.path.join(folder,ff)

        #-------------------Skip symlink-------------------
        if os.path.islink(abpathff):
            print('\n# <_doBackup>: Skip symlink %s' %ff)
            continue

        #----------------Skip if non-exists----------------
        if not os.path.exists(abpathff):
            print("\n# <_doBackup>: Can't find %s, skip." %ff)
            continue

        #--------------Copy to backup folder--------------
        copy2(abpathff,bk_path,True)

    return

def doBackup():

    #---------Create backup dir if not exists---------
    bk_path=os.path.expanduser(BKDIR)
    if not os.path.exists(bk_path):
        os.makedirs(bk_path)
    _doBackup(bk_path)

    return

def doUpdateGit():

    git_path=os.path.expanduser(GIT_DIR)
    if not os.path.exists(git_path):
        raise Exception("Git repo not found.")
    _doBackup(git_path)

    return

def doInstall():

    git_dir=os.path.expanduser(GIT_DIR)

    if not os.path.exists(git_dir):
        raise Exception("Git repo not found.")

    doBackup()

    #----------------Loop through files----------------
    for ff in FILES:
        home_path=os.path.join('~',ff)
        home_path=os.path.expanduser(home_path)
        git_path=os.path.join(git_dir,ff)

        #--------Create link from git repo to HOME--------
        if os.path.exists(home_path):
            try:
                os.remove(home_path)
            except:
                shutil.rmtree(home_path)

        os.symlink(git_path,home_path)
        print('# <install>: Create symlink to $HOME: %s' %home_path)

    return

def doRestore():

    bkdir=os.path.expanduser(BKDIR)
    if not os.path.exists(bkdir):
        raise Exception("Backup folder not found.")

    #----------------Loop through files----------------
    for ff in FILES:

        bkpath=os.path.join(bkdir,ff)
        if not os.path.exists(bkpath):
            print('\n# <install>: File/folder not found in backup, skip', ff)
            continue

        #------------------Remove symlink------------------
        home_path=os.path.join('~',ff)
        home_path=os.path.expanduser(home_path)

        if not os.path.islink(home_path):
            print('\n# <install>: File/folder not a link, skip', ff)
            continue

        if os.path.exists(home_path) and os.path.islink(home_path):
            os.remove(home_path)

        #---------------Copy backup to home---------------
        if os.path.isdir(bkpath):
            copy2(bkpath,home_path,True)
        else:
            shutil.copy(bkpath,home_path)

        print('\n# <restore>: Restored backup file: %s' %ff)

if __name__=='__main__':

    parser=argparse.ArgumentParser(description='Manage dotfiles in home folder.')

    group=parser.add_mutually_exclusive_group()
    group.add_argument('-i','--install',help='backup and install from git',
            action='store_true')
    group.add_argument('-r','--restore',help='restore backup',
            action='store_true')
    group.add_argument('-b','--backup',help='backup dotfiles, default to ~/.dotfiles_bk',
            action='store_true')
    group.add_argument('-u','--update_git',help='update to git folder',
            action='store_true')

     try:
        args=parser.parse_args()
        funcs={args.backup: doBackup,
               args.install: doInstall,
               args.restore: doRestore,
               args.update_git: doUpdateGit}
        if not any(funcs.keys()):
            print("Provide an action option")
            parser.print_help()
            sys.exit(1)
    except:
        sys.exit(1)

    for actionii, funcii in funcs.items():
        if actionii:
            funcii()

Breakdown of the script

At the top of the script I define a GIT_DIR global parameter to specify the location of the git repo, and a BKDIR parameter to specify the back-up folder. After that, the FILES list stores a list of relative (wrt to the HOME folder) paths of the dotfiles/dotfolders that we are about to sync.

In the __main__ part of the script, I use argparse to create a parser that accepts and parses input arguments from the command line. Into the parser I add a mutually exclusive group by:

group=parser.add_mutually_exclusive_group()

This makes the arguments added into the group mutually exclusive to each other so that you cannot combine them in a single command. The arguments are:

  • -i or --install: backup existing dotfiles in the system to a dedicated location (BKDIR) and create symlinks from the git repo (GIT_DIR) back to the original locations.
  • -r or --restore: restore backed-up dotfiles from the backup location (BKDIR) to their original locations.
  • -b or --backup: back up dotfiles to a dedicated location (BKDIR).
  • -u or --update_git: use the selected dotfiles in the system to update the files in the git repo.

A schematic plot showing these processes are given in Figure 1 below.

Figure 1. Schematic for the git workflow in syncing dotfiles.

The single dash line + single letter (e.g. -i), and a double dash + longer word (e.g. --install) combination for command line options is a Linux convention. All the options are boolean types, as indicated by the action='store_true' keyword argument. Because these are mutually exclusive options, only one of them is allowed to be true, then the corresponding function defined in funcs will be executed. For instance, if the -i option is given:

python git_dotfiles.py -i

args.install will be set to True, then the corresponding value, doInstall, in the funcs dict, will be called in

for actionii, funcii in funcs.items():
	if actionii:
	funcii()

The different actions are implemented in separate functions. E.g. doBackup() for the -b option, doInstall() for -i, etc.. There are also some code reuses in these functions, for instance, doInstall() calls doBackup() to back-up the dotfiles before moving them over into the git repo, and the doBackup(), doUpdateGit() functions both call _doBackup() to achieve the file back-up operation. A few other things to note:

  • the copy2() function copies a source file/folder to a target location, with slightly different treatments for files and folders.
  • similar to copy2(), move2() is doing a moving operation.
  • the file/folder paths defined in FILES are all relative to the HOME folder, therefore, when performing actual file/folder operations, we need to translate them back to absolute paths. This is done, e.g. in _doBackup():
abpathff = os.path.join('~', ff)
abpathff = os.path.expanduser(abpathff)
  • When doing back-ups, I’m skipping the symlinks, otherwise the symlinks will overwrite the actual back-up files if the back-up function is run for a 2nd time. Checking whether a file/folder is a symlink is achieved using:
os.path.islink(abpathff)
  • to create a symlink from a source location src to a target location target, one can use
os.symlink(src, target)

The workflow

Initial setup

I make my customizations by editing the dotfiles, then select those that I’d like to back-up and sync, and add their relative paths to the FILES list in the script. Then create a local git repo and a back-up folder, then put their absolute paths to the GIT_DIR and BKDIR parameters, respectively. Then run the script as

python git_dotfiles.py -u

to copy the selected dotfiles to the git repo. Finally,

python git_dotfiles.py -i

to create symlinks from the git repo back to the original locations.

Make changes to the dotfiles and sync the changes

I can edit the dotfiles as if they were at their original locations, for instance vim ~/.bashrc. What actually happens is that it will follow the symlink from ~/.bashrc to ~/.dotfiles/.bashrc, where ~/.dotfiles is the git repo. The changes are synced via git.

Get the dotfiles on a new system installation

On a different Linux machine, or a fresh installation, I can git clone the git repo from the remote to a local repo named again as ~/.dotfiles. Then execute:

python git_dotfiles.py -i

This will symlink my synced dotfiles to their destination locations so that I get the same set of customization as on my old working environment.

Related Posts

Leave a Reply