* TODO: Add ideas produced from Cauldron 2013.

ChangeLog

@@ -1,3 +1,7 @@
+2016-02-15  Ben Elliston  <bje@gnu.org>
+
+	* TODO: Add ideas produced from Cauldron 2013.
+
 2016-02-15  Ben Elliston  <bje@gnu.org>
 
 	* TODO: Update.

TODO

@@ -1,5 +1,3 @@
-Last updated $Date: 2004/02/08 14:02:07 $
-
 Bigger items
 ============
 
@@ -9,3 +7,140 @@ Bigger items
 * Add more support for target boards and RTOSes.
 * Use the new expect terminal support for an "escape codes" API.
 * Use expectk and write a GUI testing API, complete with record/playback.
+
+
+
+Date: Thu, 29 Aug 2013 19:42:07 +0200
+From: Jan Kratochvil <jan.kratochvil@redhat.com>
+To: dejagnu@gnu.org
+Subject: dejagnu-2.0 feature wishlist (from Cauldron 2013)
+
+Hi,
+
+I haven't found any discussion here about the features in hypothetical
+dejagnu-2.0, as presented by Rob Savoye at Cauldron 2013.
+
+I wrote some scripts on top of DejaGNU but I think at least some of the
+functionality could be integrated into DejaGNU itself.  It depends whether
+dejagnu-2.0 scope will remain the same or whether DejaGNU should be used
+together with tools like buildbot or whether dejagnu-2.0 will integrate some
+of the buildbot-like functionality (multi-note continuous runs).
+
+Maybe there exists something similar already?  Originally I wrote it only for
+myself but I see nowadays such tool may be useful for more people.
+
+Former announcement of my scripts:
+	https://sourceware.org/ml/archer/2010-q3/msg00194.html
+URLs are no longer valid, the files can be found now at:
+	git clone git://git.jankratochvil.net/nethome
+	(that is my whole $HOME, not just the testsuite scripts)
+The primary script 'hammock' is at:
+	http://git.jankratochvil.net/?p=nethome.git;a=blob;f=bin/hammock
+
+Essential fixup of current DejaGNU:
+------------------------------------------------------------------------------
+--orphanripper: It is used by default, normal DejaGNU scripts do not track
+their spawned children which share fds 0/1/2 (stdio).  This means some such
+children are due to *.exp code bugs occasionally leftover running forever.  As
+they have their fds still open the testsuite with output redirected somewhere
+will lock up at the end.  Some runaway processes also hog CPU for 100%.  The
+following utility identifies runaway processes by using custom pty for them
+and kills them at the end of testsuite run:
+	http://pkgs.fedoraproject.org/cgit/gdb.git/plain/gdb-orphanripper.c
+It sure should be better integrated in DejaGNU somehow.
+
+Features outside of the current scope of DejaGNU:
+------------------------------------------------------------------------------
+--distro: Testing in various OSes.  My script implement it based on chroot
+(Fedora/RHEL has tool 'mock' for it), it has some performance/management
+advantages but it has to (1) run all OSes with the same kernel, (2) mock
+supports only Fedora/RHEL OSes, (3) it can run only x86_64/i386 arch this way.
+The real solution should be multi-node (so that it can also support non-x86*
+testing), for x86* it would be commonly using VMs.  But it still could support
+even mock/chroot as it runs without the hassle of disk images.
+
+--component: Pre-set remote repositories for download of gdb/binutils/gcc etc.
+I want to run my patches on top of clean tree, not in some existing directory
+which may have leftover files forgotten to be checked into repository etc.
+Understandably it also supports local repository caches.
+--srcrpm is similar, it builds tree from a prepared archive - I should be able
+to provide also .src.rpm (or .tar.gz) to run the test for.
+--branch asks for example for branch "gdb_7_6-branch" from the repository.
+
+--file: Provide custom patches for the newly built tree.
+
+--target: Provide a list of custom configure --target options.  This could be
+more general such as to provide any custom configure options.
+
+--parallel: Parallelization of multiple build+testsuite runs, not just
+parallelization of the testsuite run part.
+If I ask to build 40x binutils with 40 different targets I may want to do
+run it in parallel (like with make -j8).
+
+Convenience:
+------------------------------------------------------------------------------
+--gdbserver, --valgrind, --bfd32, --gdbindex, --dwz, --dwarf=X, --stabs:
+Various pre-set options.  One can configure it by hand but it is too difficult
+for daily use, for example for --dwz it means for GDB
+	runtest CC_FOR_TARGET=/bin/sh\ $PWD/../contrib/cc-with-tweaks.sh\ -m\ gcc CXX_FOR_TARGET=/bin/sh\ $PWD/../contrib/cc-with-tweaks.sh\ -m\ g++ ...
+	(plus also GNATMAKE_FOR_TARGET, GO_FOR_TARGET and GO_LD_FOR_TARGET)
+For --valgrind it means other cryptic options like:
+	RUNTESTFLAGS=--target_board=valgrind
+
+Incomplete racy reads ("read1"):
+------------------------------------------------------------------------------
+https://sourceware.org/bugzilla/show_bug.cgi?id=12649
+GDB testsuite contains (yes, it still contains them) various racy cases:
+  gdb_test_multiple "set dprintf-style agent" $msg {
+      -re "warning: Target cannot run dprintf commands.*" {
+It commonly works as when expect does the read() syscall all the GDB output is
+ready.  But occasionally the next testcase FAILs.  This is because
+occasionally only part of the output gets read by the read() syscall, regex
+gets matched but the final $gdb_prompt is not discarded - and the leftover
+$gdb_prompt corrupts the next testcase below.  Sure the fix is:
+  gdb_test_multiple "set dprintf-style agent" $msg {
+    -re "warning: Target cannot run dprintf commands.*\r\n$gdb_prompt $" {
+
+There is LD_PRELOAD *.so file in the Bug above to reproduce these cases
+reliably.  There is also a reproducer of different kind of bugs ("writew")
+although those do not happen so often AFAIK.  This functionality could be
+better integrated into DejaGNU.
+
+(Sure the primary problem is that the testsuite should not use regex matching
+and it should use generic GDB MI output parser.  But that is a problem that
+only a few GDB features have implemented the GDB MI interface.)
+
+Diffing of results:
+------------------------------------------------------------------------------
+http://git.jankratochvil.net/?p=nethome.git;a=blob;f=bin/diffgdb
+ * GDB has various known FAILs.  They should be but they are not KFAILed or
+   XFAILed.  (On recent Fedora I see there are only 23 of them but on CentOS-5
+   there is 1063 of them.)
+ * One is only interested in introduced regressions so one needs to diff two
+   *.sum files.  Looking again and again at the same known FAILing cases is
+   not productive.
+ * During diff one is not interested for example in newly PASSing testcases.
+   One also is not interested in FAIL->PASS cases.  One is definitely
+   interested in PASS->FAIL regressions.  New FAILing testcases are also
+   interesting.
+Therefore the script above does a filtering of the diff results.  It parses
+DejaGNU *.sum output although DejaGNU did already knew them internally.
+
+The script also filters out unstable/racy results.  This may be outside of the
+scope but in fact an unimplemented feature would be to provide statistics on
+unstable results (so one can fix those) if I run the same build+testsuite many
+times.
+
+Not yet implemented: Finding a regression common reason:
+------------------------------------------------------------------------------
+I run 73 testsuite runs daily - primarily GDB in different OSes, for each OS
+its x86_64 and i686 variant, for x86_64 OS also in -m32 mode.
+If there happens a general regression I get 73 times PASS->FAIL result.  That
+is not too convenient to filter out other changes out of the 73 regressions.
+Moreover sometimes the regression affects for example only 32-bit OSes
+- therefore there will be only about 24 PASS->FAILs and I have to figure out
+in which testsuite combinations they happen.
+In other cases the regression happens for example only on (older) RHELs and
+not on Fedoras but that again means about 6 PASS->FAIL cases.
+There would be nice some summary that this PASS->FAIL occured in testsuite
+runs in directories rhel* and it did not occur in directories fedora* etc.