What's the easiest way to deal with project configuration files?

It's very common to have at least one configuration file in any project. Every time I share project with git, I have the same problem with:

  • sensitive information (each developer have different DB passwords etc)
  • task specific information (when developer works on certain task where one needs to change some settings)

Obviously, configs have to be ignored somehow to prevent developer specific data to flood main repository. Now there are several ways I used to use, each with some flaws:

  • .gitignore the config files
    • the most basic way
    • when developer clones repo, config file is missing and one has to find out where configs were recreate them
  • config file isn't ignored. It contains some dummy info and each developer either untracks and puts it to his .git/info/exclude or set git update-index --assume-unchanged ... to the file
    • files are available for anyone who clones repo
    • it contains advanced techniques which could confuse people that work with git for the first time
    • when someone commits config files by accident, it won't allow people to pull/fetch (as excludes don't work the same way as .gitignore)
  • distribute config files suffixed with, for example, _original while having the real files in .gitignore. Each developer then renames files to real names
    • files are available for anyone who clones repo
    • one has to search for all configs throughout application and rename them

Are there any other, possibly, better ways to handle this? I suspect I'm missing something, some plugin at least.

Answers


Filter drivers are the "automatic" way of implementing option 3, as detailed in "when you have secret key in your project, how can pushing to GitHub be possible?":

The smudge script will, on checkout:

  • detect the right config files to modify
  • fetch the information needed (best kept outside any Git repo) and will replace the template values by the actual one.

From there the developers can make any kind of modification they want to those config files. It won't matter, because the clean script will, on commit, restore the content of that file to its original (template) value. No accidental push there.


The way we did it on the last project i worked on was to have a master config file that loaded a users local config file if it was present which could overwrite the defaults set in the master if specified and declared its own config info if not present in master. The local file was added to gitignore. That way all common stuff could all be shared and some config always present and each developer can modify their local.


Since it took me a while to figure out a working solution with @VonC 's hints, here's a full example of how to ignore passwords with a git clean filter in an Objective-C header file.

  1. Assume you have a default config script named Config.h containing this

    // Change "12345" to your password
    #define kPass @"12345"
    #define kConstant 42
    
  2. Create a script hidepass.sh that matches the critical line(s) and prints the default line instead

    #!/bin/sh
    awk '{ if (/#define kPass/) print "#define kPass @\"12345\""; else print $0; }'
    exit 0
    
  3. Make the script executable and add it to the repo

  4. Tell git to filter your config file by adding this line to .gitattributes (also add .gitattributes to your repo)

    Config.h filter=hidepass
    
  5. Tell git to use the hidepass.sh script for the hidepass filter during clean:

    git config filter.hidepass.clean ./hidepass.sh
    

That's it, you can now change the password in Config.h but git will not commit this change because it always replaces that line with the default line on checkin.

This is a quick solution for a one-line password, you can go crazy and e.g. end the lines to be ignored with a special string and check for lines with that string.


In projects I have been, we have a default config, and developers have their own config at a particular location outside version control ( convention over configuration ) - the values from latter are used to override those in the former.

We started using encryption for sensitive details in the config: Handling passwords in production config for automated deployment

In case of git, you can look at git attributes filter attribute to do both the replacement of local values and decryption of sensitive values in an automated way.

You can also have submodules which have say the production.yml and with restricted access to the submodule repo.


One thing I can think of is similar to method 2 and method 1 above. Where you have a directory where you store complex things that are unique to the site such as directories for config files, user uploaded files, etc.

You keep just the config file itself out of version control, but then you have a dummy copy that is named slightly differently from what the site actually uses and this file contains detailed instructions as to the config parameters and to make a renamed copy.

For example, lets say you have a "site_profile" directory. In this directory you create a file called "README.settings.php" along with a "files" directory that contains user uploaded files (from both admin and front-end users). All of this is under version control.

However, the site will run its settings from "settings.php" which won't exist in here. But if you were to rename the "README.settings.php" to "settings.php" then you would have the config file you need (after you put in your custom settings of course).

This will allow you to tell the other developers what they need out of their config file while keeping your own config file out of the mix. Just set your config file to be ignored or never do a blanket commit for that directory and lower.

Its what we do with Drupal sites where I work and it works really well.


For the two cases you mention:

  • sensitive information (each developer have different DB passwords etc)

    Write your scripts to so that these sensitive files are stored in the developer's home directory, rather than in the project directory.

  • task specific information (when developer works on certain task where one needs to change some settings)

    I usually have the default settings checked into the repository, and then when you prepare your commits, you can easily check if you've modified any of those settings, and revert them before you commit.


I saved my local config changes to a git stash. I use git stash apply rather than pop, so that I never remove it from the stash queue. That way I can use reset --hard whenever I feel like it.


Having no experience with Git, but having dealt with the same issues in other contexts (Hibernate login creds in a Spring JUnit suite, or ANT scripts), for anything that is user-specific or depends on a user's local environment:

  1. Have a README listing those dependencies and how to configure your local env to run the script.
  2. Replace those dependencies in your conf with calls to system properties, or some framework way to pull in external values. In an ANT script, for instance, you can replace strings with <sysproperty key="jdbc.username" value="${jdbc.username}"/>, where jdbc.username is a system variable (that you can set in Eclipse).

Developers can check in whatever they want, because the conf is user-agnostic.


A lot of good options listed. A few more options to mention:

  • Add all configs to .gitignore. Then have a separate git project, with the just the configuration files. (Keep this GIT #2 in a totally different folder.) In this GIT #2, make a few scripts similar to apply-config /path/to/my_git1, save-config /path/to/my_git1 which copy in, or save configuration files for the main GIT repo.
    • I prefer this to using submodules. I've found working with submodules cumbersome across a large team.
    • You can then track configurations, or use multiple GIT repos for things like production settings, debug settings, more optimized settings, etc.
    • Uses one tool (GIT) that everyone understands.
  • I like the idea of looking in the home folder for sensitive things like DB connection info. (Mentioned by @avh)
  • For devOps, you may want to consider things like Ansible / Salt / Chef / Puppet for automating deployments and settings.

Another very common option, popularized by The Twelve-Factor App, is to not have any configuration variables hardcoded into files, but rather load them from environmental variables. This way, the files in repository don't need any special treatment.


Need Your Help

Saving a variable data to disk

c# .net xna

Is it possible to save a variable from C# to disk so that you are able to use it later in another instance of your project?