Automatic Wikification of Configuration Files
In the company I work for, most documentation is maintained in Word format. Except mine. I have a dokuwiki and am thankfully allowed this exception as I am the only Linuxer in the company. Since Windows systems need external documentation (being hindered by the absence of commentable text configuration files), there is a policy that all configuration data needs to be explicitly documented. I hate that idea, since documentation is always outdated, and documenting configuration changes doubles the work that needs to be done.
After finding out that dokuwiki has a command line interface, I implemented a mechanism that can run from cron and keeps wiki pages of configuration files up to date on an automated basis.
On the target systems (the ones that need to have their configuration documented), I installed a script which generates a tarball containing the relevant configuration files on standard output. For more sensitive data, the script could do some basic sanitazion such as removing passwords from the configuration files before putting them into the tarball.
On the system running the cron job, the following script runs:
#!/bin/bash
set -e
set -C
set -u
JOBNAME="update-wiki-firewall-config"
DWPAGE="php4 /usr/share/dokuwiki/bin/dwpage.php"
TMPBASE="/tmp"
SSHID="$HOME/.ssh/$USER-passphraseless-$(hostname)"
REMOTEHOST="remote.fqdn.example"
REMOTEJOB="pull-netfilter-init-config-tar"
SHAREDIR="$HOME/share/$JOBNAME"
WIKINAMESPACE="pointer:to:namespace"
start='/*BEGIN generated firewall rules /'
end='/*END generated firewall rules /'
insert='/*INSERT generated firewall rules /'
umask 077
if ! TMPDIR="$(mktemp -d $TMPBASE/fwconfig.XXXXXXXXXX)"; then
echo >&2 "ERR: cannot create temp dir in $TMPBASE"
exit 1
fi
cd $TMPDIR
mkdir remote
ssh -i $SSHID root@$REMOTEHOST $REMOTEJOB | \\
tar --extract --gzip --file - --directory remote
mkdir workdir
touch workdir/rulelistfile
echo "$start" >> workdir/rulelistfile
for fwfile in $(find $TMPDIR/remote/rules/up -type f | sort); do
MTIME="$(stat --format="%y" $fwfile)"
PROSEEXPLANATION="$(< $fwfile sed -n '/^#::#/{s/^#::#[[:space:]]*//;p;q;}')"
if [ -z "$PROSEEXPLANATION" ]; then
PROSEEXPLANATION="netfilter-init internal code"
fi
rm -f workdir/newwikipage
< $SHAREDIR/wikipagetemplate \\
sed \\
-e "s|PLHfilenamePLH|$(basename $fwfile)|" \\
-e "s|PLHproseexplanationPLH|$PROSEEXPLANATION|" \\
-e "s|PLHmtimePLH|$MTIME|" \\
-e "/PLHcontentsPLH/r$fwfile" \\
-e "s/PLHcontentsPLH//" \\
> workdir/newwikipage
WIKIPAGENAME="$WIKINAMESPACE:regeln:$(basename $fwfile .rul)"
$DWPAGE checkout $WIKIPAGENAME workdir/wikipage
if ! cmp --quiet workdir/wikipage workdir/newwikipage; then
cp workdir/newwikipage workdir/wikipage
$DWPAGE -m "new contents imported from $fwfile" commit workdir/wikipage $WIKIPAGENAME
fi
echo " [[$WIKIPAGENAME|$(basename $fwfile)]] - $PROSEEXPLANATION" >> workdir/rulelistfile
done
echo "$end" >> workdir/rulelistfile
$DWPAGE checkout $WIKINAMESPACE workdir/contentspage.txt
< workdir/contentspage.txt \\
sed -e "\|^${start//\\\*/\\\*}\\$|,\|^${end//\\\*/\\\*}\\$|{d;}" | \\
sed -e "\|^${insert//\\\*/\\\*}\\$|rworkdir/rulelistfile" \\
> workdir/newcontentspage.txt
if ! cmp --quiet workdir/contentspage.txt workdir/newcontentspage.txt; then
cp workdir/newcontentspage.txt workdir/contentspage.txt
$DWPAGE -m "new contents page" commit workdir/contentspage.txt $WIKINAMESPACE
fi
cd /
rm -rf "$TMPDIR"
Here are the two template files for wiki page and contents page:
====== Firewallregeln PLHfilenamePLH ======
Last changed on target system: PLHmtimePLH
PLHproseexplanationPLH
<code>
PLHcontentsPLH
</code>
===== Dump of firewall rules =====
/*INSERT generated firewall rules /
/*BEGIN generated firewall rules /
/*END generated firewall rules */
The index page template relys on the HiddenComment dokuwiki plugin to be installed, or the placeholders show up in the rendered wiki page. The code uses the first comment line marked with #::# to generate a short explanation of the file being pulled into the wiki. dwpage.php is a fully-fledged command line interface to dokuwiki, committed new pages are fulled versioned as if entered through the wiki web front end. This way, the documentation in the wiki is guaranteed to be current and to fit the configuration found on the live system - it is directly pulled from the live system.
Yes, rendering of the code is currently suboptimal. Working on it.
Trackbacks
No Trackbacks
Comments
Display comments as Linear | Threaded
Anonymous on :
You might find this a lot easier with something like ikiwiki, which natively supports flat files (with a revision control system) as the backend. Furthermore, with some work on the syntax highlighting support, you could probably keep /etc in a revision control system and point ikiwiki at that.
Marc 'Zugschlus' Haber on :
How can it be any easier? Dokuwiki uses flat files, and it has an easily useable CLI.
I have never actually managed to put /etc into a VCS since none of the VCSses I have tried deals adequately with symlinks, file owners and access privileges. What's the purpose of shadow passwords when they're world readable in the VCS repository?
Anonymous on :
I don't know anything about docuwiki, and the way you described it, it sounded like docuwiki didn't. Many wikis can technically have a CLI, if you use a manual script that applies the guts of the rendering engine; the way you described dwpage.php, it sounded like such a script.
As for "how can it get any easier", with ikiwiki, you wouldn't necessarily need to generate a wiki file containing the configuration file contents; you could just reference the data file directly with the inline or syntax directives.
Finally, regarding the use of a VCS for /etc: most of the modern revision control systems deal fine with symlinks, at least. As for permissions and file owners, some VCSes track them, and for those that don't, a simple commit hook would let you save and restore permissions. Finally, you don't check shadow into a VCS, or you make the repository privileged. Few other files have permission issues, though; in fact, on my system, only /etc/shadow, /etc/gshadow, and /etc/sudoers do. sudoers doesn't actually contain anything that needs to remain private, and the shadow files can either go in a privileged repository or just remain unversioned.
Marc 'Zugschlus' Haber on :
First, let me say that I really appreciate this discussion.
dwpage.php is actually much like a version control system: It has a checkout and a commit option, and between checkout and commit, the file in the wiki is locked so that nobody else can commit. Just take a look at the script which uses dwpage.php to put the new files into the wiki.
I can understand that file inclusion in the wiki syntax might ease things, but in the current project, I'd like to have file descriptions in the rendered page outside of the code as well, and thus need to generate the template page anyway. And while I am doing this, I can easily dump the file contents as well in the page itself. One less file, and I think this is actually easier to understand than an inline include.
Subversion, the VCS of my choice, is documented to handle symlinks just fine, but I never tried it before and do not feel too comfortable with trying it with something as vital as /etc of a live system. Privileged repositories kind of rule out working copies for "normal users", which would make configuration handling even more a "root only" affair than it is on a conservatively maintained system, and my systems have a lot more confidential files in /etc, such as password files for services (be it as a server or as a client, htaccess, e-mail passwords etc) or cryptographic keys (private keys for SSL stuff).
"a simple commit hook" is the standard answer to all these problems, but when one looks closer, there does not seem to be even proof-of-concept code for such hooks. Or, the hooks cannot access the repository or cannot create their own working copy. And I'm too lazy to work out the details of these myself, I firmly believe that this should be done by people who know the VCS a lot better than I do.
Anonymous on :
You might also look at IsiSetup.