summaryrefslogtreecommitdiff
path: root/Makeconfig
diff options
context:
space:
mode:
authorJoseph Myers <joseph@codesourcery.com>2014-03-07 03:23:34 +0000
committerJoseph Myers <joseph@codesourcery.com>2014-03-07 03:23:34 +0000
commit54ee292da7a65acf72ac5655777555786899119c (patch)
tree979f95d5cd5bc482c456c2fabc26960429365722 /Makeconfig
parenta00f522af84ac3f18ff97341858a6c91a7986ad8 (diff)
conformtest: split up running of tests from makefile.
This patch changes the logic by which header conformance tests are run so that the makefiles run the tests for each (standard, header) pair separately rather than as a single test (there are 518 such pairs being tested at present). Since these tests are slow and previously couldn't be run in parallel, this obviously speeds up the whole test run significantly when using parallel testing. There are other benefits. These tests can now be marked as expected to fail at the level of (standard, header) pairs, meaning that regressions introduced by a header change are much more likely to be spotted (of course, such a regression could be a bug in the header or in the expectations, most of which have not been properly checked against the relevant standards). The patch introduces lists in conform/Makefile of the headers to test for each standard, with a new (quick) test for each standard that the list agrees with the set of headers for which there are in fact nonempty expectations for that standard. This means that each of the 518 pairs being tested is actually something meaningful to test rather than a null test. I also hope to use these lists of headers in various standards in future tests for namespace violations where a function in a standard is implemented to call a non-reserved name for a function outside that standard. Although there are a lot of tests now marked as expected to fail, I expect most of those to be easy to fix (whether with fixes to the expectations, the headers or both - if fixing a header bug, of course file it in Bugzilla first); only a limited number are likely to reflect actual missing features in glibc. Tested x86_64 (GCC 4.7 branch). It's quite possible some failures vary from platform to platform, in which case either additional XFAILs can be added here, or platform-specific header bugs (if applicable) the failures show up can be fixed. I made XFAILs unconditional for tests that should only fail for GCC 4.6 and earlier; we could make the GCC version available in the makefiles and condition these, but simply moving the XFAILing to a finer granularity seems a clear improvement on the previous state of the whole of conformtest being XFAILed. * Makeconfig (test-xfail-name): New variable. (evaluate-test): Use $(test-xfail-name) instead of $(@F:.out=) to compute variable name for expected failures. * conform/Makefile (conformtest-headers-data): New variable. (conformtest-standards): Likewise. (conformtest-headers-ISO): Likewise. (conformtest-headers-ISO99): Likewise. (conformtest-headers-ISO11): Likewise. (conformtest-headers-POSIX): Likewise. (conformtest-headers-XPG3): Likewise. (conformtest-headers-XPG4): Likewise. (conformtest-headers-UNIX98): Likewise. (conformtest-headers-XOPEN2K): Likewise. (conformtest-headers-POSIX2008): Likewise. (conformtest-headers-XOPEN2K8): Likewise. (conformtest-header-list-base): Likewise. (conformtest-header-list-tests): Likewise. (conformtest-header-base): Likewise. (conformtest-header-tests): Likewise. (tests-special): Add $(conformtest-header-list-tests). If [$(fast-check) && !$(cross-compiling)], add $(conformtest-header-tests) instead of $(objpfx)run-conformtest.out. (generated): Add $(conformtest-header-list-base). If [$(fast-check) && !$(cross-compiling)], add $(conformtest-header-base). Remove previous setting. ($(conformtest-header-list-tests)): New target. (test-xfail-run-conformtest): Remove variable. ($(objpfx)run-conformtest.out): Remove target. (test-xfail-ISO11/complex.h/conform): New variable. (test-xfail-ISO11/stdalign.h/conform): Likewise. (test-xfail-ISO11/stdnoreturn.h/conform): Likewise. (test-xfail-XPG3/varargs.h/conform): Likewise. (test-xfail-XPG4/varargs.h/conform): Likewise. (test-xfail-UNIX98/varargs.h/conform): Likewise. (test-xfail-XPG4/ndbm.h/conform): Likewise. (test-xfail-UNIX98/ndbm.h/conform): Likewise. (test-xfail-XOPEN2K/ndbm.h/conform): Likewise. (test-xfail-XOPEN2K8/ndbm.h/conform): Likewise. (test-xfail-XPG3/fcntl.h/conform): Likewise. (test-xfail-XPG3/ftw.h/conform): Likewise. (test-xfail-XPG3/grp.h/conform): Likewise. (test-xfail-XPG3/langinfo.h/conform): Likewise. (test-xfail-XPG3/limits.h/conform): Likewise. (test-xfail-XPG3/pwd.h/conform): Likewise. (test-xfail-XPG3/search.h/conform): Likewise. (test-xfail-XPG3/signal.h/conform): Likewise. (test-xfail-XPG3/stdio.h/conform): Likewise. (test-xfail-XPG3/stdlib.h/conform): Likewise. (test-xfail-XPG3/string.h/conform): Likewise. (test-xfail-XPG3/sys/ipc.h/conform): Likewise. (test-xfail-XPG3/sys/msg.h/conform): Likewise. (test-xfail-XPG3/sys/sem.h/conform): Likewise. (test-xfail-XPG3/sys/shm.h/conform): Likewise. (test-xfail-XPG3/sys/stat.h/conform): Likewise. (test-xfail-XPG3/sys/types.h/conform): Likewise. (test-xfail-XPG3/sys/wait.h/conform): Likewise. (test-xfail-XPG3/termios.h/conform): Likewise. (test-xfail-XPG3/time.h/conform): Likewise. (test-xfail-XPG3/unistd.h/conform): Likewise. (test-xfail-XPG4/arpa/inet.h/conform): Likewise. (test-xfail-XPG4/fcntl.h/conform): Likewise. (test-xfail-XPG4/langinfo.h/conform): Likewise. (test-xfail-XPG4/netdb.h/conform): Likewise. (test-xfail-XPG4/netinet/in.h/conform): Likewise. (test-xfail-XPG4/signal.h/conform): Likewise. (test-xfail-XPG4/stdio.h/conform): Likewise. (test-xfail-XPG4/stdlib.h/conform): Likewise. (test-xfail-XPG4/stropts.h/conform): Likewise. (test-xfail-XPG4/sys/ipc.h/conform): Likewise. (test-xfail-XPG4/sys/msg.h/conform): Likewise. (test-xfail-XPG4/sys/sem.h/conform): Likewise. (test-xfail-XPG4/sys/shm.h/conform): Likewise. (test-xfail-XPG4/sys/socket.h/conform): Likewise. (test-xfail-XPG4/sys/stat.h/conform): Likewise. (test-xfail-XPG4/sys/time.h/conform): Likewise. (test-xfail-XPG4/sys/types.h/conform): Likewise. (test-xfail-XPG4/sys/wait.h/conform): Likewise. (test-xfail-XPG4/termios.h/conform): Likewise. (test-xfail-XPG4/ucontext.h/conform): Likewise. (test-xfail-XPG4/unistd.h/conform): Likewise. (test-xfail-XPG4/utmpx.h/conform): Likewise. (test-xfail-POSIX/sched.h/conform): Likewise. (test-xfail-POSIX/signal.h/conform): Likewise. (test-xfail-POSIX/sys/wait.h/conform): Likewise. (test-xfail-POSIX/tar.h/conform): Likewise. (test-xfail-UNIX98/arpa/inet.h/conform): Likewise. (test-xfail-UNIX98/fcntl.h/conform): Likewise. (test-xfail-UNIX98/langinfo.h/conform): Likewise. (test-xfail-UNIX98/netdb.h/conform): Likewise. (test-xfail-UNIX98/netinet/in.h/conform): Likewise. (test-xfail-UNIX98/sched.h/conform): Likewise. (test-xfail-UNIX98/signal.h/conform): Likewise. (test-xfail-UNIX98/stdio.h/conform): Likewise. (test-xfail-UNIX98/stdlib.h/conform): Likewise. (test-xfail-UNIX98/stropts.h/conform): Likewise. (test-xfail-UNIX98/sys/ipc.h/conform): Likewise. (test-xfail-UNIX98/sys/msg.h/conform): Likewise. (test-xfail-UNIX98/sys/sem.h/conform): Likewise. (test-xfail-UNIX98/sys/shm.h/conform): Likewise. (test-xfail-UNIX98/sys/socket.h/conform): Likewise. (test-xfail-UNIX98/sys/time.h/conform): Likewise. (test-xfail-UNIX98/sys/wait.h/conform): Likewise. (test-xfail-UNIX98/ucontext.h/conform): Likewise. (test-xfail-UNIX98/unistd.h/conform): Likewise. (test-xfail-UNIX98/utmpx.h/conform): Likewise. (test-xfail-XOPEN2K/aio.h/conform): Likewise. (test-xfail-XOPEN2K/arpa/inet.h/conform): Likewise. (test-xfail-XOPEN2K/fcntl.h/conform): Likewise. (test-xfail-XOPEN2K/langinfo.h/conform): Likewise. (test-xfail-XOPEN2K/math.h/conform): Likewise. (test-xfail-XOPEN2K/mqueue.h/conform): Likewise. (test-xfail-XOPEN2K/netdb.h/conform): Likewise. (test-xfail-XOPEN2K/netinet/in.h/conform): Likewise. (test-xfail-XOPEN2K/semaphore.h/conform): Likewise. (test-xfail-XOPEN2K/signal.h/conform): Likewise. (test-xfail-XOPEN2K/stdarg.h/conform): Likewise. (test-xfail-XOPEN2K/stdio.h/conform): Likewise. (test-xfail-XOPEN2K/stropts.h/conform): Likewise. (test-xfail-XOPEN2K/sys/ipc.h/conform): Likewise. (test-xfail-XOPEN2K/sys/msg.h/conform): Likewise. (test-xfail-XOPEN2K/sys/sem.h/conform): Likewise. (test-xfail-XOPEN2K/sys/shm.h/conform): Likewise. (test-xfail-XOPEN2K/sys/socket.h/conform): Likewise. (test-xfail-XOPEN2K/sys/wait.h/conform): Likewise. (test-xfail-XOPEN2K/termios.h/conform): Likewise. (test-xfail-XOPEN2K/tgmath.h/conform): Likewise. (test-xfail-XOPEN2K/ucontext.h/conform): Likewise. (test-xfail-XOPEN2K/utmpx.h/conform): Likewise. (test-xfail-POSIX2008/arpa/inet.h/conform): Likewise. (test-xfail-POSIX2008/fcntl.h/conform): Likewise. (test-xfail-POSIX2008/netdb.h/conform): Likewise. (test-xfail-POSIX2008/netinet/in.h/conform): Likewise. (test-xfail-POSIX2008/signal.h/conform): Likewise. (test-xfail-POSIX2008/stropts.h/conform): Likewise. (test-xfail-POSIX2008/sys/socket.h/conform): Likewise. (test-xfail-POSIX2008/sys/wait.h/conform): Likewise. (test-xfail-XOPEN2K8/arpa/inet.h/conform): Likewise. (test-xfail-XOPEN2K8/fcntl.h/conform): Likewise. (test-xfail-XOPEN2K8/limits.h/conform): Likewise. (test-xfail-XOPEN2K8/math.h/conform): Likewise. (test-xfail-XOPEN2K8/netdb.h/conform): Likewise. (test-xfail-XOPEN2K8/netinet/in.h/conform): Likewise. (test-xfail-XOPEN2K8/signal.h/conform): Likewise. (test-xfail-XOPEN2K8/stdio.h/conform): Likewise. (test-xfail-XOPEN2K8/stropts.h/conform): Likewise. (test-xfail-XOPEN2K8/sys/ipc.h/conform): Likewise. (test-xfail-XOPEN2K8/sys/msg.h/conform): Likewise. (test-xfail-XOPEN2K8/sys/select.h/conform): Likewise. (test-xfail-XOPEN2K8/sys/sem.h/conform): Likewise. (test-xfail-XOPEN2K8/sys/shm.h/conform): Likewise. (test-xfail-XOPEN2K8/sys/socket.h/conform): Likewise. (test-xfail-XOPEN2K8/sys/time.h/conform): Likewise. (test-xfail-XOPEN2K8/sys/wait.h/conform): Likewise. (test-xfail-XOPEN2K8/termios.h/conform): Likewise. (test-xfail-XOPEN2K8/tgmath.h/conform): Likewise. (test-xfail-XOPEN2K8/utmpx.h/conform): Likewise. (conformtest-cc-flags): Likewise. ($(conformtest-header-tests): New target. * conform/check-header-lists.sh: New file. * conform/run-conformtest.sh: Remove.
Diffstat (limited to 'Makeconfig')
-rw-r--r--Makeconfig7
1 files changed, 5 insertions, 2 deletions
diff --git a/Makeconfig b/Makeconfig
index 3709c88a45..9078b29306 100644
--- a/Makeconfig
+++ b/Makeconfig
@@ -1083,11 +1083,14 @@ endif
# The name to give to a test in test results summaries.
test-name = $(strip $(patsubst %.out, %, $(patsubst $(common-objpfx)%, %, $@)))
+# Likewise, in XFAIL variable names.
+test-xfail-name = $(strip $(patsubst %.out, %, $(patsubst $(objpfx)%, %, $@)))
+
# Command to output a test status line (such as PASS: test-name). If
-# test-xfail-$(@F:.out=) has a nonempty value, the status will be
+# test-xfail-$(test-xfail-name) has a nonempty value, the status will be
# XPASS or XFAIL rather than PASS or FAIL.
evaluate-test = $(..)scripts/evaluate-test.sh $(test-name) $$? \
- $(if $(test-xfail-$(@F:.out=)),true,false) \
+ $(if $(test-xfail-$(test-xfail-name)),true,false) \
> $(common-objpfx)$(test-name).test-result
endif # Makeconfig not yet included