riscv/qemu - qemu - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Philippe Mathieu-Daudé	1a1e889f35	buildsys: Bump known good meson version to v1.2.3 We need meson v1.2.3 to build QEMU on macOS Sonoma. It also builds fine all our CI jobs (as tested by also bumping "accepted" in pythondeps.toml), so let's use it as our "good enough" packaged wheel. Resolves: https://gitlab.com/qemu-project/qemu/-/issues/1939 Suggested-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org> Message-ID: <20231109160504.93677-2-philmd@linaro.org> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>	2 years ago
Paolo Bonzini	0a88ac9662	Revert "mkvenv: work around broken pip installations on Debian 10" Debian 10 has Python 3.7, so it is not possible to use it anymore now that Python 3.8 is required. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>	3 years ago
Paolo Bonzini	3e4b6b0ad9	mkvenv: assume presence of importlib.metadata importlib.metadata is included in Python 3.8, so there is no need to fallback to either importlib-metadata or pkgresources when generating console script shims. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>	3 years ago
Paolo Bonzini	ca056f4499	Python: Drop support for Python 3.7 Debian 10 is not anymore a supported distro, since Debian 12 was released on June 10, 2023. Our supported build platforms as of today all support at least 3.8 (and all of them except for Ubuntu 20.04 support 3.9): openSUSE Leap 15.5: 3.6.15 (3.11.2) CentOS Stream 8: 3.6.8 (3.8.13, 3.9.16, 3.11.4) CentOS Stream 9: 3.9.17 (3.11.4) Fedora 37: 3.11.4 Fedora 38: 3.11.4 Debian 11: 3.9.2 Debian 12: 3.11.2 Alpine 3.14, 3.15: 3.9.16 Alpine 3.16, 3.17: 3.10.10 Ubuntu 20.04 LTS: 3.8.10 Ubuntu 22.04 LTS: 3.10.12 NetBSD 9.3: 3.9.13* FreeBSD 12.4: 3.9.16 FreeBSD 13.1: 3.9.18 OpenBSD 7.2: 3.9.17 Note: NetBSD does not appear to have a default meta-package, but offers several options, the lowest of which is 3.7.15. However, "python39" appears to be a pre-requisite to one of the other packages we request in tests/vm/netbsd. Since it is safe under our supported platform policy, bump our minimum supported version of Python to 3.8. The two most interesting features to have by default include: - the importlib.metadata module, whose lack is responsible for over 100 lines of code in mkvenv.py - improvements to asyncio, for example asyncio.CancelledError inherits from BaseException rather than Exception In addition, code can now use the assignment operator ':=' Because mypy now learns about importlib.metadata, a small change to mkvenv.py is needed to pass type checking. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>	3 years ago
Paolo Bonzini	c03f57fd5b	Revert "tests: Use separate virtual environment for avocado" This reverts commit `e8e4298fea`. ensuregroup allows to specify both the acceptable versions of avocado, and a locked version to be used when avocado is not installed as a system pacakge. This lets us install avocado in pyvenv/ using "mkvenv.py" and reuse the distro package on Fedora and CentOS Stream (the only distros where it's available). ensuregroup's usage of "(>=..., <=...)" constraints when evaluating the distro package, and "==" constraints when installing it from PyPI, makes it possible to avoid conflicts between the known-good version and a package plugins included in the distro. This is because package plugins have "==" constraints on the version that is included in the distro, and, using "pip install avocado==88.1" on a venv that includes system packages will result in an error: avocado-framework-plugin-varianter-yaml-to-mux 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible. avocado-framework-plugin-result-html 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible. But at the same time, if the venv does not include a system distribution of avocado then we can install a known-good version and stick to LTS releases. Resolves: https://gitlab.com/qemu-project/qemu/-/issues/1663 Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>	3 years ago
Paolo Bonzini	edc2107895	python: use vendored tomli Debian only introduced tomli in the bookworm release. Use a vendored wheel to avoid requiring a package that is only in bullseye-backports and is also absent in Ubuntu 20.04. While at it, fix an issue in the vendor.py scripts which does not add a newline after each package and hash. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>	3 years ago
Paolo Bonzini	71ed611cd4	python: mkvenv: add ensuregroup command Introduce a new subcommand that retrieves the packages to be installed from a TOML file. This allows being more flexible in using the system version of a package, while at the same time using a known-good version when installing the package. This is important for packages that sometimes have backwards-incompatible changes or that depend on specific versions of their dependencies. Compared to JSON, TOML is more human readable and easier to edit. A parser is available in 3.11 but also available as a small (12k) package for older versions, tomli. While tomli is bundled with pip, this is only true of recent versions of pip. Of all the supported OSes pretty much only FreeBSD has a recent enough version of pip while staying on Python <3.11. So we cannot use the same trick that is in place for distlib. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>	3 years ago
Paolo Bonzini	0f1ec0705b	python: mkvenv: introduce TOML-like representation of dependencies We would like to place all Python dependencies in the same file, so that we can add more information without having long and complex command lines. The plan is to have a TOML file with one entry per package, for example [avocado] avocado-framework = { accepted = "(>=88.1, <93.0)", installed = "88.1", canary = "avocado" } Each TOML section will thus be a dictionary of dictionaries. Modify mkvenv.py's workhorse function, _do_ensure, to already operate on such a data structure. The "ensure" subcommand is modified to separate the depspec into a name and a version part, and use the result (plus the --diagnose argument) to build a dictionary for each command line argument. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>	3 years ago
Paolo Bonzini	67b9a83daf	python: mkvenv: tweak the matching of --diagnose to depspecs Move the matching between the "absent" array and dep_specs[0] inside the loop, preparing for the possibility of having multiple canaries among the installed packages. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>	3 years ago
Paolo Bonzini	47a90a51a9	mkvenv: always pass locally-installed packages to pip Let pip decide whether a new version should be installed or the current one is okay. This ensures that the virtual environment is updated (either upgraded or downgraded) whenever a new version of a package is requested. The hardest part here is figuring out if a package is installed in the venv (which also has to be done twice to account for the presence of either setuptools in Python <3.8, or importlib in Python >=3.8). Suggested-by: Peter Maydell <peter.maydell@linaro.org> Cc: John Snow <jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>	3 years ago
Paolo Bonzini	d37c21b5fb	mkvenv: pass first missing package to diagnose() If sphinx is present but the theme is not, mkvenv will print an inaccurate diagnostic: ERROR: Could not find a version that satisfies the requirement sphinx-rtd-theme>=0.5.0 (from versions: none) ERROR: No matching distribution found for sphinx-rtd-theme>=0.5.0 'sphinx>=1.6.0' not found: • Python package 'sphinx' version '5.3.0' was found, but isn't suitable. • mkvenv was configured to operate offline and did not check PyPI. Instead, ignore the packages that were found to be present, and report an error based on the first absent package. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>	3 years ago
Paolo Bonzini	c673f3d0fe	mkvenv: replace distlib.database with importlib.metadata/pkg_resources importlib.metadata is just as good as distlib.database and a bit more battle-proven for "egg" based distributions, and in fact that is exactly why mkvenv.py is not using distlib.database to find entry points: it simply does not work for eggs. The only disadvantage of importlib.metadata is that it is not available by default before Python 3.8, so we need a fallback to pkg_resources (again, just like for the case of finding entry points). Do so to fix issues where incorrect egg metadata results in a JSONDecodeError. While at it, reuse the new _get_version function to diagnose an incorrect version of the package even if importlib.metadata is not available. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>	3 years ago
Paolo Bonzini	3b087f79a4	meson: require 0.63.0 This version allows cleanups in modinfo collection, but they only work with Ninja 1.9.x and 1.8.x is still supported. It also supports the equivalent of QEMU's --static option to configure. The wheel file is bumped to 0.63.3, the last release in the 0.63 branch. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>	4 years ago
Paolo Bonzini	02312f1af1	mkvenv: mark command as required This is only available in Python 3.7+. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20230511035435.734312-26-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>	3 years ago
John Snow	2274817f6c	python: add vendor.py utility This is a teeny-tiny script that just downloads any packages we want to vendor from PyPI and stores them in qemu.git/python/wheels/. If I'm hit by a meteor, it'll be easy to replicate what I have done in order to udpate the vendored source. We don't really care which python runs it; it exists as a meta-utility with no external dependencies and we won't package or install it. It will be monitored by the linters/type checkers, though; so it's guaranteed safe on python 3.6+. Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20230511035435.734312-15-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>	3 years ago
John Snow	c804962617	mkvenv: work around broken pip installations on Debian 10 This is a workaround intended for Debian 10, where the debian-patched pip does not function correctly if accessed from within a virtual environment. We don't support Debian 10 as a build platform any longer, though we do still utilize it for our build-tricore-softmmu CI test. It's also possible that this bug might appear on other derivative platforms and this workaround may prove useful. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20230511035435.734312-11-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>	3 years ago
John Snow	f1ad527ff5	mkvenv: avoid ensurepip if pip is installed Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>	3 years ago
John Snow	68ea6d17fe	mkvenv: use pip's vendored distlib as a fallback distlib is usually not installed on Linux distribution, but it is vendored into pip. Because the virtual environment has pip via ensurepip, we can piggy-back on pip's vendored version. This could break if they move our cheese in the future, but the fix would be simply to require distlib. If it is debundled, as it is on msys, it is simply available directly. Signed-off-by: John Snow <jsnow@redhat.com> [Move to toplevel. - Paolo] Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>	3 years ago
John Snow	928348949d	mkvenv: add console script entry point generation When creating a virtual environment that inherits system packages, script entry points (like "meson", "sphinx-build", etc) are not re-generated with the correct shebang. When you are inside of the venv, this is not a problem, but if you are outside of it, you will not have a script that engages the virtual environment appropriately. Add a mechanism that generates new entry points for pre-existing packages so that we can use these scripts to run "meson", "sphinx-build", "pip", unambiguously inside the venv. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20230511035435.734312-9-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>	3 years ago
John Snow	4695a22e9a	mkvenv: add --diagnose option to explain "ensure" failures This is a routine that is designed to print some usable info for human beings back out to the terminal if/when "mkvenv ensure" fails to locate or install a package during configure time, such as meson or sphinx. Since we are requiring that "meson" and "sphinx" are installed to the same Python environment as QEMU is configured to build with, this can produce some surprising failures when things are mismatched. This method is here to try and ease that sting by offering some actionable diagnosis. Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20230511035435.734312-8-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>	3 years ago
John Snow	c5538eed12	mkvenv: add ensure subcommand This command is to be used to add various packages (or ensure they're already present) into the configure-provided venv in a modular fashion. Examples: mkvenv ensure --online --dir "${source_dir}/python/wheels/" "meson>=0.61.5" mkvenv ensure --online "sphinx>=1.6.0" mkvenv ensure "qemu.qmp==0.0.2" It's designed to look for packages in three places, in order: (1) In system packages, if the version installed is already good enough. This way your distribution-provided meson, sphinx, etc are always used as first preference. (2) In a vendored packages directory. Here I am suggesting qemu.git/python/wheels/ as that directory. This is intended to serve as a replacement for vendoring the meson source for QEMU tarballs. It is also highly likely to be extremely useful for packaging the "qemu.qmp" package in source distributions for platforms that do not yet package qemu.qmp separately. (3) Online, via PyPI, *only when "--online" is passed. This is only ever used as a fallback if the first two sources do not have an appropriate package that meets the requirement. The ability to build QEMU and run tests completely offline* is not impinged. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20230511035435.734312-7-jsnow@redhat.com> [Use distlib to lookup distributions. - Paolo] Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>	3 years ago
John Snow	dee01b827f	mkvenv: add nested venv workaround Python virtual environments do not typically nest; they may inherit from the top-level system packages or not at all. For our purposes, it would be convenient to emulate "nested" virtual environments to allow callers of the configure script to install specific versions of python utilities in order to test build system features, utility version compatibility, etc. While it is possible to install packages into the system environment (say, by using the --user flag), it's nicer to install test packages into a totally isolated environment instead. As detailed in https://www.qemu.org/2023/03/24/python/, Emulate a nested venv environment by using .pth files installed into the site-packages folder that points to the parent environment when appropriate. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20230511035435.734312-6-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>	3 years ago
John Snow	a9dbde71da	mkvenv: add better error message for broken or missing ensurepip Debian debundles ensurepip for python; NetBSD debundles pyexpat but ensurepip needs pyexpat. Try our best to offer a helpful error message instead of just failing catastrophically. Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20230511035435.734312-5-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>	3 years ago
John Snow	dd84028ff9	python: add mkvenv.py This script will be responsible for building a lightweight Python virtual environment at configure time. It works with Python 3.6 or newer. It has been designed to: - work offline, no PyPI required. - work quickly, The fast path is only ~65ms on my machine. - work robustly, with multiple fallbacks to keep things working. - work cooperatively, using system packages where possible. (You can use your distro's meson, no problem.) Due to its unique position in the build chain, it exists outside of the installable python packages in-tree and must be runnable without any third party dependencies. Under normal circumstances, the only dependency required to execute this script is Python 3.6+ itself. The script is faster by several seconds when setuptools and pip are installed in the host environment, which is probably the case for a typical multi-purpose developer workstation. In the event that pip/setuptools are missing or not usable, additional dependencies may be required on some distributions which remove certain Python stdlib modules to package them separately: - Debian may require python3-venv to provide "ensurepip" - NetBSD may require py310-expat to provide "pyexpat" * (* Or whichever version is current for NetBSD.) Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20230511035435.734312-4-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>	3 years ago

24 Commits (e1999904a960c33b68fedf26dfb7b8e00abab8f2)