Skip to content

Automatic Wikification of Configuration Files

In the company I work for, most documentation is maintained in Word format. Except mine. I have a dokuwiki and am thankfully allowed this exception as I am the only Linuxer in the company. Since Windows systems need external documentation (being hindered by the absence of commentable text configuration files), there is a policy that all configuration data needs to be explicitly documented. I hate that idea, since documentation is always outdated, and documenting configuration changes doubles the work that needs to be done.

After finding out that dokuwiki has a command line interface, I implemented a mechanism that can run from cron and keeps wiki pages of configuration files up to date on an automated basis.

On the target systems (the ones that need to have their configuration documented), I installed a script which generates a tarball containing the relevant configuration files on standard output. For more sensitive data, the script could do some basic sanitazion such as removing passwords from the configuration files before putting them into the tarball.

On the system running the cron job, the following script runs:


set -e
set -C
set -u


DWPAGE="php4 /usr/share/dokuwiki/bin/dwpage.php"
start='/*BEGIN generated firewall rules /'
end='/*END generated firewall rules /'
insert='/*INSERT generated firewall rules /'

umask 077

if ! TMPDIR="$(mktemp -d $TMPBASE/fwconfig.XXXXXXXXXX)"; then
        echo >&2 "ERR: cannot create temp dir in $TMPBASE"
        exit 1

mkdir remote
 tar --extract --gzip --file - --directory remote

mkdir workdir
touch workdir/rulelistfile
echo "$start" >> workdir/rulelistfile

for fwfile in $(find $TMPDIR/remote/rules/up -type f | sort); do
  MTIME="$(stat --format="%y" $fwfile)"
  PROSEEXPLANATION="$(< $fwfile sed -n '/^#::#/{s/^#::#[[:space:]]*//;p;q;}')"
  if [ -z "$PROSEEXPLANATION" ]; then
    PROSEEXPLANATION="netfilter-init internal code"
  rm -f workdir/newwikipage
  < $SHAREDIR/wikipagetemplate \\
    sed \\
        -e "s|PLHfilenamePLH|$(basename $fwfile)|" \\
        -e "s|PLHproseexplanationPLH|$PROSEEXPLANATION|" \\
        -e "s|PLHmtimePLH|$MTIME|" \\
        -e "/PLHcontentsPLH/r$fwfile" \\
        -e "s/PLHcontentsPLH//" \\
  > workdir/newwikipage
  WIKIPAGENAME="$WIKINAMESPACE:regeln:$(basename $fwfile .rul)"
  $DWPAGE checkout $WIKIPAGENAME workdir/wikipage
  if ! cmp --quiet workdir/wikipage workdir/newwikipage; then
    cp workdir/newwikipage workdir/wikipage
    $DWPAGE -m "new contents imported from $fwfile" commit workdir/wikipage $WIKIPAGENAME
  echo "   [[$WIKIPAGENAME|$(basename $fwfile)]] - $PROSEEXPLANATION" >> workdir/rulelistfile
echo "$end" >> workdir/rulelistfile

$DWPAGE checkout $WIKINAMESPACE workdir/contentspage.txt
< workdir/contentspage.txt \\
  sed -e "\|^${start//\\\*/\\\*}\\$|,\|^${end//\\\*/\\\*}\\$|{d;}" | \\
  sed -e "\|^${insert//\\\*/\\\*}\\$|rworkdir/rulelistfile" \\
  > workdir/newcontentspage.txt

if ! cmp --quiet workdir/contentspage.txt workdir/newcontentspage.txt; then
  cp workdir/newcontentspage.txt workdir/contentspage.txt
  $DWPAGE -m "new contents page" commit workdir/contentspage.txt $WIKINAMESPACE

cd /
rm -rf "$TMPDIR"
Yes, this needs some more documentation and generalization. But it's just a proof of concept that needs some polishing before being put into real life use.

Here are the two template files for wiki page and contents page:

====== Firewallregeln PLHfilenamePLH ======

Last changed on target system: PLHmtimePLH


===== Dump of firewall rules =====

/*INSERT generated firewall rules /
/*BEGIN generated firewall rules /

/*END generated firewall rules */

The index page template relys on the HiddenComment dokuwiki plugin to be installed, or the placeholders show up in the rendered wiki page. The code uses the first comment line marked with #::# to generate a short explanation of the file being pulled into the wiki. dwpage.php is a fully-fledged command line interface to dokuwiki, committed new pages are fulled versioned as if entered through the wiki web front end. This way, the documentation in the wiki is guaranteed to be current and to fit the configuration found on the live system - it is directly pulled from the live system.

Yes, rendering of the code is currently suboptimal. Working on it.


No Trackbacks


Display comments as Linear | Threaded

Anonymous on :

You might find this a lot easier with something like ikiwiki, which natively supports flat files (with a revision control system) as the backend. Furthermore, with some work on the syntax highlighting support, you could probably keep /etc in a revision control system and point ikiwiki at that.

Marc 'Zugschlus' Haber on :

How can it be any easier? Dokuwiki uses flat files, and it has an easily useable CLI.

I have never actually managed to put /etc into a VCS since none of the VCSses I have tried deals adequately with symlinks, file owners and access privileges. What's the purpose of shadow passwords when they're world readable in the VCS repository?

Anonymous on :

I don't know anything about docuwiki, and the way you described it, it sounded like docuwiki didn't. Many wikis can technically have a CLI, if you use a manual script that applies the guts of the rendering engine; the way you described dwpage.php, it sounded like such a script.

As for "how can it get any easier", with ikiwiki, you wouldn't necessarily need to generate a wiki file containing the configuration file contents; you could just reference the data file directly with the inline or syntax directives.

Finally, regarding the use of a VCS for /etc: most of the modern revision control systems deal fine with symlinks, at least. As for permissions and file owners, some VCSes track them, and for those that don't, a simple commit hook would let you save and restore permissions. Finally, you don't check shadow into a VCS, or you make the repository privileged. Few other files have permission issues, though; in fact, on my system, only /etc/shadow, /etc/gshadow, and /etc/sudoers do. sudoers doesn't actually contain anything that needs to remain private, and the shadow files can either go in a privileged repository or just remain unversioned.

Marc 'Zugschlus' Haber on :

First, let me say that I really appreciate this discussion.

dwpage.php is actually much like a version control system: It has a checkout and a commit option, and between checkout and commit, the file in the wiki is locked so that nobody else can commit. Just take a look at the script which uses dwpage.php to put the new files into the wiki.

I can understand that file inclusion in the wiki syntax might ease things, but in the current project, I'd like to have file descriptions in the rendered page outside of the code as well, and thus need to generate the template page anyway. And while I am doing this, I can easily dump the file contents as well in the page itself. One less file, and I think this is actually easier to understand than an inline include.

Subversion, the VCS of my choice, is documented to handle symlinks just fine, but I never tried it before and do not feel too comfortable with trying it with something as vital as /etc of a live system. Privileged repositories kind of rule out working copies for "normal users", which would make configuration handling even more a "root only" affair than it is on a conservatively maintained system, and my systems have a lot more confidential files in /etc, such as password files for services (be it as a server or as a client, htaccess, e-mail passwords etc) or cryptographic keys (private keys for SSL stuff).

"a simple commit hook" is the standard answer to all these problems, but when one looks closer, there does not seem to be even proof-of-concept code for such hooks. Or, the hooks cannot access the repository or cannot create their own working copy. And I'm too lazy to work out the details of these myself, I firmly believe that this should be done by people who know the VCS a lot better than I do.

Anonymous on :

You might also look at IsiSetup.

Add Comment

Markdown format allowed
Enclosing asterisks marks text as bold (*word*), underscore are made via _word_.
Standard emoticons like :-) and ;-) are converted to images.
E-Mail addresses will not be displayed and will only be used for E-Mail notifications.
Form options