riscv/qemu - qemu - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Fam Zheng	c1958e9d54	docker: Improved image checksum When a base image locally defined by QEMU, such as in the debian images, is updated, the dockerfile checksum mechanism in docker.py still skips updating the derived image, because it only looks at the literal content of the dockerfile, without considering changes to the base image. For example we have a recent fix `e58c1f9b35` that fixed debian-win64-cross by updating its base image, debian8-mxe, but due to above "feature" of docker.py the image in question is automatically NOT rebuilt unless you add NOCACHE=1. It is noticed on Shippable: https://app.shippable.com/github/qemu/qemu/runs/541/2/console because after the fix is merged, the error still occurs, and the log shows the container image is, as explained above, not updated. This is because at the time docker.py was written, there wasn't any dependencies between QEMU's docker images. Now improve this to preprocess any "FROM qemu:*" directives in the dockerfiles while doing checksum, and inline the base image's dockerfile content, recursively. This ensures any changes on the depended _QEMU_ images are taken into account. This means for external images that we expect to retrieve from docker registries, we still do it as before. It is not perfect, because registry images can get updated too. Technically we could substitute the image name with its hex ID as obtained with $(docker images $IMAGE --format="{{.Id}}"), but --format is not supported by RHEL 7, so leave it for now. Reported-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Signed-off-by: Fam Zheng <famz@redhat.com> Message-Id: <20171103131229.4737-1-famz@redhat.com> Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Signed-off-by: Fam Zheng <famz@redhat.com>	9 years ago
Alex Bennée	6fe3ae3f19	docker: docker.py make --no-cache skip checksum test If you invoke with NOCACHE=1 we pass --no-cache in the argv to docker.py but may still not force a rebuild if the dockerfile checksum hasn't changed. By testing for its presence we can force builds without having to manually remove the docker image. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20170725133425.436-5-alex.bennee@linaro.org> Signed-off-by: Fam Zheng <famz@redhat.com>	9 years ago
Stefan Hajnoczi	c2d3189667	docker.py: Python 2.6 argparse compatibility Add the scripts/ directory to sys.path so Python 2.6 will be able to import argparse. Cc: Fam Zheng <famz@redhat.com> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> Acked-by: John Snow <jsnow@redhat.com> Acked-by: Fam Zheng <famz@redhat.com> Message-id: 20170825155732.15665-3-stefanha@redhat.com Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>	9 years ago
Fam Zheng	0b95ff72cb	docker.py: Improve subprocess exit code handling A few error handlings are missing because we ignore the subprocess exit code, for example "docker build" errors are currently ignored. Introduce _do_check() aside the existing _do() method and use it in a few places. Signed-off-by: Fam Zheng <famz@redhat.com> Message-Id: <20170712075528.22770-3-famz@redhat.com> Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Signed-off-by: Fam Zheng <famz@redhat.com>	9 years ago
Fam Zheng	58bf7b6d8c	docker.py: Drop infile parameter The **kwargs can do this just well. Signed-off-by: Fam Zheng <famz@redhat.com> Message-Id: <20170712075528.22770-2-famz@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Signed-off-by: Fam Zheng <famz@redhat.com>	9 years ago
Philippe Mathieu-Daudé	438d116872	docker: rebuild image if 'extra files' checksum does not match Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>	9 years ago
Philippe Mathieu-Daudé	4c84f662c2	docker: add --include-files argument to 'build' command Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>	9 years ago
Philippe Mathieu-Daudé	2499ee9fad	docker: let _copy_with_mkdir() sub_path argument be optional Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>	9 years ago
Philippe Mathieu-Daudé	06cc355171	tests/docker: support proxy / corporate firewall if ftp_proxy/http_proxy/https_proxy standard environment variables available, pass them to the docker daemon to build images. this is required when building behind corporate proxy/firewall, but also help when using local cache server (ie: apt/yum). Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Message-Id: <20170306205520.32311-1-f4bug@amsat.org> Reviewed-by: Fam Zheng <famz@redhat.com> Signed-off-by: Fam Zheng <famz@redhat.com>	9 years ago
Alex Bennée	414a8ce57e	tests/docker: add basic user mapping support Currently all docker builds are done by exporting a tarball to the docker container and running the build as the containers root user. Other use cases are possible however and it is possible to map a part of users file-system to the container. This is useful for example for doing cross-builds of arbitrary source trees. For this to work smoothly the container needs to have a user created that maps cleanly to the host system. This adds a -u option to the docker script so that: DEB_ARCH=armhf DEB_TYPE=stable ./tests/docker/docker.py build \ -u --include-executable=arm-linux-user/qemu-arm \ debian:armhf ./tests/docker/dockerfiles/debian-bootstrap.docker Will build a container that can then be run like: docker run --rm -it -v /home/alex/lsrc/qemu/risu.git/:/src \ --user=alex:alex -w /src/ debian:armhf \ sh -c "make clean && ./configure -s && make" All docker containers built will add the current user unless explicitly disabled by specifying NOUSER when invoking the Makefile: make docker-image-debian-armhf-cross NOUSER=1 Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Fam Zheng <famz@redhat.com> Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Message-Id: <20170220105139.21581-2-alex.bennee@linaro.org> Signed-off-by: Fam Zheng <famz@redhat.com>	9 years ago
Fam Zheng	97cba1a1d1	docker: Terminate instances at SIGTERM and SIGHUP Signed-off-by: Fam Zheng <famz@redhat.com> Message-Id: <1474429768-25027-9-git-send-email-famz@redhat.com> Reviewed-by: Daniel P. Berrange <berrange@redhat.com>	10 years ago
Eduardo Habkost	0679f98b40	docker: Handle exceptions when looking for docker command When trying to run docker tests on a host without the docker command, we get the following Python backtrace: $ make docker-test-quick@centos6 V=1 .../qemu/tests/docker/docker.py build qemu:centos6 .../qemu/tests/docker/dockerfiles/centos6.docker Traceback (most recent call last): File ".../qemu/tests/docker/docker.py", line 339, in <module> sys.exit(main()) File ".../qemu/tests/docker/docker.py", line 336, in main return args.cmdobj.run(args, argv) File ".../qemu/tests/docker/docker.py", line 231, in run dkr = Docker() File ".../qemu/tests/docker/docker.py", line 98, in __init__ self._command = _guess_docker_command() File ".../qemu/tests/docker/docker.py", line 41, in _guess_docker_command stdout=DEVNULL, stderr=DEVNULL) == 0: File "/usr/lib64/python2.7/subprocess.py", line 523, in call return Popen(popenargs, kwargs).wait() File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__ errread, errwrite) File "/usr/lib64/python2.7/subprocess.py", line 1343, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory .../qemu/tests/docker/Makefile.include:47: recipe for target 'docker-image-centos6' failed make: [docker-image-centos6] Error 1 Change _guess_docker_command() to handle OSError exceptions raised by subprocess.call(), so we will keep looking for other commands and print a better error message. New output will be: $ make docker-test-quick@centos6 V=1 .../qemu/tests/docker/docker.py build qemu:centos6 .../qemu/tests/docker/dockerfiles/centos6.docker Traceback (most recent call last): File ".../qemu/tests/docker/docker.py", line 343, in <module> sys.exit(main()) File ".../qemu/tests/docker/docker.py", line 340, in main return args.cmdobj.run(args, argv) File ".../qemu/tests/docker/docker.py", line 235, in run dkr = Docker() File ".../qemu/tests/docker/docker.py", line 102, in __init__ self._command = _guess_docker_command() File ".../qemu/tests/docker/docker.py", line 49, in _guess_docker_command commands_txt) Exception: Cannot find working docker command. Tried: docker sudo -n docker .../qemu/tests/docker/Makefile.include:47: recipe for target 'docker-image-centos6' failed make: * [docker-image-centos6] Error 1 Signed-off-by: Eduardo Habkost <ehabkost@redhat.com> Message-Id: <1474369559-16903-1-git-send-email-ehabkost@redhat.com> [exceptions.OSError -> OSError and drop the import. - Fam] Signed-off-by: Fam Zheng <famz@redhat.com>	10 years ago
Sascha Silbe	f8042deafa	docker: silence debootstrap when --quiet is given If we silence docker when --quiet is given, we should also silence the .pre script (i.e. debootstrap). Only discards stdout, so some diagnostics (e.g. from git clone) are still printed. Most of the verbose output is gone however and this way we still have a chance to see error messages. Signed-off-by: Sascha Silbe <silbe@linux.vnet.ibm.com> Message-Id: <1473192351-601-9-git-send-email-silbe@linux.vnet.ibm.com> Signed-off-by: Fam Zheng <famz@redhat.com>	10 years ago
Sascha Silbe	c977257045	docker.py: don't hang on large docker output Unlike Popen.communicate(), subprocess.call() doesn't read from the stdout file descriptor. If the child process produces more output than fits into the pipe buffer, it will block indefinitely. If we don't intend to consume the output, just send it straight to /dev/null to avoid this issue. Signed-off-by: Sascha Silbe <silbe@linux.vnet.ibm.com> Reviewed-by: Janosch Frank <frankja@linux.vnet.ibm.com> Message-Id: <1473192351-601-2-git-send-email-silbe@linux.vnet.ibm.com> Signed-off-by: Fam Zheng <famz@redhat.com>	10 years ago
Fam Zheng	4b08af6019	docker: Add "images" subcommand to docker.py This is a wrapper for the 'docker images' command. Signed-off-by: Fam Zheng <famz@redhat.com> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Message-id: 1468934445-32183-9-git-send-email-famz@redhat.com	10 years ago
Alex Bennée	6e733da676	tests/docker/docker.py: add update operation This adds a new operation to the docker script to allow updating of binaries in an existing container. This is because it would be inefficient to re-build the whole container just for an update to the QEMU binary. To update the executable run: ./tests/docker/docker.py update \ debian:armhf ./arm-linux-user/qemu-arm Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-id: 1468934445-32183-6-git-send-email-famz@redhat.com Signed-off-by: Fam Zheng <famz@redhat.com>	10 years ago
Alex Bennée	920776ea5e	tests/docker/docker.py: check and run .pre script The docker script will now search for an associated $dockerfile.pre script which gets run in the same build context as the dockerfile will be. This is to support pre-seeding the build context before running the docker build. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-id: 1468934445-32183-4-git-send-email-famz@redhat.com Signed-off-by: Fam Zheng <famz@redhat.com>	10 years ago
Alex Bennée	504ca3c208	tests/docker/docker.py: support --include-executable When passed the path to a binary we copy it and any linked libraries (if it is dynamically linked) into the docker build context. These can then be included by a dockerfile with the line: # Copy all of context into container ADD . / This is mainly intended for setting up foreign architecture docker images which use qemu-$arch to do cross-architecture linux-user execution. It also relies on the host and guest file-system following reasonable multi-arch layouts so the copied libraries don't clash with the guest ones. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-id: 1468934445-32183-3-git-send-email-famz@redhat.com Signed-off-by: Fam Zheng <famz@redhat.com>	10 years ago
Alex Bennée	a9f8d03891	tests/docker/docker.py: docker_dir outside build Instead of letting the build_image create the temporary working dir we move the creation to the build command. This is preparation for the later patches where additional files can be added to the build context before the build step is run. We also ensure we remove the build context after we are done (mkdtemp doesn't do this automatically for you). Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Fam Zheng <famz@redhat.com> Message-id: 1468934445-32183-2-git-send-email-famz@redhat.com Signed-off-by: Fam Zheng <famz@redhat.com>	10 years ago
Fam Zheng	4485b04be9	tests: Add utilities for docker testing docker.py is added with a number of useful subcommands to manager docker images and instances for QEMU docker testing. Subcommands are: run: A wrapper of "docker run" (or "sudo -n docker run" if necessary), which takes care of killing and removing the running container at SIGINT. clean: Tear down all the containers including inactive ones that are started by docker_run. build: Compare an image from given dockerfile and rebuild it if they're different. Signed-off-by: Fam Zheng <famz@redhat.com> Message-id: 1464755128-32490-2-git-send-email-famz@redhat.com	10 years ago

20 Commits (c1958e9d54c8de85ceda7c96b302b75a2f67b4e1)