Verify dgit using official git tests

It is possible to use the official git tests to verify dgit both in terms of correctness and completeness. This guide covers how to set up the test environment and run the tests. Note that a subset of these tests are included in the Travis CI builds so that you can track which suites are already passing and which ones need to be skipped at any given time.

There is a Unix bash script run-official-tests that will clone the official git repository at the supported version tag (2.8.0 at the time of writing), builds it and runs make with the arguments you provide to the script. For example, to run the basic test suite you can run ./run-official-tests.sh t0000-basic.sh to run just that one suite. If you want to run all suites you can provide no arguments at all and it will go through the hundreds of test suites. If you provide to make the "-i" flag then it will run the specified test suites and not fail on the first error. Note that you can run this script multiple times and it will short-cut much of the setup the second and subsequent times.

Once you run the script once the test suites can be found in the official-git/t directory. Each script is written as a bash shell script that you can read to discover how they work. Note that make only gives rather limited output. You can get more detailed output from a test suite by executing it directly. There are details in the README in that folder. As a simple example, you can run sh ./t0000-basic.sh -v to get verbose output. The "-i" and "-d" flags can also be very helpful when trying to track down the cause of test failures.

Many of the tests are going to fail for now. The output can be very noisy. Here is a command that will run all of the zero-level test suites and show only the pass/fail with a small description of the test.

./run-official-tests.sh | grep -E '^not ok|^ok'

There are times when you need to figure out what goes wrong with one or more tests. You can force dgit into trace mode and output the trace information to a trace file by setting the DGIT_TRACE environment variable. Note that the shell scripts customize the environment and usually clean this variable before running the tests. You can get around that by using a shell script in place of the dgit binary placed in official-git/git directory. Remove the git (ie. dgit) binary in that directory, copy the git-init script in its place, rework it so that it points to the dgit binary location and remove the "init" subcommand from the arguments. Before it invokes dgit put a line that exports and sets the DGIT_TRACE environment variable to output the tracing information. The tracing information will give you the working directory and all of the command-line arguments used to invoke dgit, which is super helpful when trying to reproduce the problem.

There are times when dgit hangs running the suites, especially with the master branch. If you notice that it gets stuck for a long time you can run killall dgit to free it to run the remainder of the tests.

Results

As of July 2018, the git test suite was run on the t0*.sh scripts, which cover much of the core git functionality. The results were that dgit hanged three times (bugs have already been raised for these), passed 676 tests and failed 2059 tests. The themes for many of the failures is the hang, broken dgit config functionality, missing subcommands (e.g. check-ignore, tag, archive) and missing crlf mapping functionality.

Note that parts of the git test suite use a separate test-helper command built with git to test git internals and don't invoke dgit. These will spuriously report success.

Here is a list of commands that were touched in the tests but not yet implemented:

Unknown git command am.
Unknown git command archive.
Unknown git command check-attr.
Unknown git command credential-cache.
Unknown git command credential.
Unknown git command for-each-ref.
Unknown git command ls-remote.
Unknown git command mv.
Unknown git command reflog.
Unknown git command revisions.
Unknown git command rm.
Unknown git command script.
Unknown git command sh-i18n.
Unknown git command show.
Unknown git command stripspace.
Unknown git command submodule.
Unknown git command tag.
Unknown git command var.
Unknown git command worktree.

Here are some flags that are missing or have discrepancies:

git --bare init <somerepo> # Note the --bare is before the init subcommand
git init --shared=...
git init --separate-git-dir
git -c key=value # Override a config value only for the length of this command
git diff --exit-code
git diff-tree -M
git ls-files --eol
git ls-files -X
git hash-object --no-filters