riscv/qemu - qemu - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Paolo Bonzini	ca056f4499	Python: Drop support for Python 3.7 Debian 10 is not anymore a supported distro, since Debian 12 was released on June 10, 2023. Our supported build platforms as of today all support at least 3.8 (and all of them except for Ubuntu 20.04 support 3.9): openSUSE Leap 15.5: 3.6.15 (3.11.2) CentOS Stream 8: 3.6.8 (3.8.13, 3.9.16, 3.11.4) CentOS Stream 9: 3.9.17 (3.11.4) Fedora 37: 3.11.4 Fedora 38: 3.11.4 Debian 11: 3.9.2 Debian 12: 3.11.2 Alpine 3.14, 3.15: 3.9.16 Alpine 3.16, 3.17: 3.10.10 Ubuntu 20.04 LTS: 3.8.10 Ubuntu 22.04 LTS: 3.10.12 NetBSD 9.3: 3.9.13* FreeBSD 12.4: 3.9.16 FreeBSD 13.1: 3.9.18 OpenBSD 7.2: 3.9.17 Note: NetBSD does not appear to have a default meta-package, but offers several options, the lowest of which is 3.7.15. However, "python39" appears to be a pre-requisite to one of the other packages we request in tests/vm/netbsd. Since it is safe under our supported platform policy, bump our minimum supported version of Python to 3.8. The two most interesting features to have by default include: - the importlib.metadata module, whose lack is responsible for over 100 lines of code in mkvenv.py - improvements to asyncio, for example asyncio.CancelledError inherits from BaseException rather than Exception In addition, code can now use the assignment operator ':=' Because mypy now learns about importlib.metadata, a small change to mkvenv.py is needed to pass type checking. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>	3 years ago
Paolo Bonzini	3d7b89748a	python: bump minimum requirements so they are compatible with 3.12 There are many Python 3.12 issues right now, but a particularly problematic one when debugging them is that one cannot even use minreqs.txt in a Python 3.12 virtual environment to test with locked package versions. Bump the mypy and wrapt versions to fix this, while remaining within the realm of versions compatible with Python 3.7. This requires a workaround for a mypy false positive qemu/qmp/qmp_tui.py:350: error: Non-overlapping equality check (left operand type: "Literal[Runstate.DISCONNECTING]", right operand type: "Literal[Runstate.IDLE]") [comparison-overlap] where mypy does not realize that self.disconnect() could change the value of self.runstate. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>	3 years ago
Paolo Bonzini	7b4b98c46c	python: bump some of the dependencies The version of pyflakes that is listed in python/tests/minreqs.txt breaks on Python 3.8 with the following message: AttributeError: 'FlakesChecker' object has no attribute 'CONSTANT' Now that we do not support EOL'd Python versions anymore, we can update to newer, fixed versions. It is a good time to do so, before Python packages start dropping support for Python 3.7 as well! The new mypy is also a bit smarter about which packages are actually being used, so remove the now-unnecessary sections from setup.cfg. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20230511035435.734312-27-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>	3 years ago
Paolo Bonzini	5591b74511	Python: Drop support for Python 3.6 Python 3.6 was EOL 2021-12-31. Newer versions of upstream libraries have begun dropping support for this version and it is becoming more cumbersome to support. Avocado-framework and qemu.qmp each have their own reasons for wanting to drop Python 3.6, but won't until QEMU does. Versions of Python available in our supported build platforms as of today, with optional versions available in parentheses: openSUSE Leap 15.4: 3.6.15 (3.9.10, 3.10.2) CentOS Stream 8: 3.6.8 (3.8.13, 3.9.16) CentOS Stream 9: 3.9.13 Fedora 36: 3.10 Fedora 37: 3.11 Debian 11: 3.9.2 Alpine 3.14, 3.15: 3.9.16 Alpine 3.16, 3.17: 3.10.10 Ubuntu 20.04 LTS: 3.8.10 Ubuntu 22.04 LTS: 3.10.4 NetBSD 9.3: 3.9.13* FreeBSD 12.4: 3.9.16 FreeBSD 13.1: 3.9.16 OpenBSD 7.2: 3.9.16 Note: Our VM tests install 3.9 explicitly for FreeBSD and 3.10 for NetBSD; the default for "python" or "python3" in FreeBSD is 3.9.16. NetBSD does not appear to have a default meta-package, but offers several options, the lowest of which is 3.7.15. "python39" appears to be a pre-requisite to one of the other packages we request in tests/vm/netbsd. pip, ensurepip and other Python essentials are currently only available for Python 3.10 for NetBSD. CentOS and OpenSUSE support parallel installation of multiple Python interpreters, and binaries in /usr/bin will always use Python 3.6. However, the newly introduced support for virtual environments ensures that all build steps that execute QEMU Python code use a single interpreter. Since it is safe to under our supported platform policy, bump our minimum supported version of Python to 3.7. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Message-Id: <20230511035435.734312-24-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>	3 years ago
John Snow	c5538eed12	mkvenv: add ensure subcommand This command is to be used to add various packages (or ensure they're already present) into the configure-provided venv in a modular fashion. Examples: mkvenv ensure --online --dir "${source_dir}/python/wheels/" "meson>=0.61.5" mkvenv ensure --online "sphinx>=1.6.0" mkvenv ensure "qemu.qmp==0.0.2" It's designed to look for packages in three places, in order: (1) In system packages, if the version installed is already good enough. This way your distribution-provided meson, sphinx, etc are always used as first preference. (2) In a vendored packages directory. Here I am suggesting qemu.git/python/wheels/ as that directory. This is intended to serve as a replacement for vendoring the meson source for QEMU tarballs. It is also highly likely to be extremely useful for packaging the "qemu.qmp" package in source distributions for platforms that do not yet package qemu.qmp separately. (3) Online, via PyPI, *only when "--online" is passed. This is only ever used as a fallback if the first two sources do not have an appropriate package that meets the requirement. The ability to build QEMU and run tests completely offline* is not impinged. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20230511035435.734312-7-jsnow@redhat.com> [Use distlib to lookup distributions. - Paolo] Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>	3 years ago
John Snow	dd84028ff9	python: add mkvenv.py This script will be responsible for building a lightweight Python virtual environment at configure time. It works with Python 3.6 or newer. It has been designed to: - work offline, no PyPI required. - work quickly, The fast path is only ~65ms on my machine. - work robustly, with multiple fallbacks to keep things working. - work cooperatively, using system packages where possible. (You can use your distro's meson, no problem.) Due to its unique position in the build chain, it exists outside of the installable python packages in-tree and must be runnable without any third party dependencies. Under normal circumstances, the only dependency required to execute this script is Python 3.6+ itself. The script is faster by several seconds when setuptools and pip are installed in the host environment, which is probably the case for a typical multi-purpose developer workstation. In the event that pip/setuptools are missing or not usable, additional dependencies may be required on some distributions which remove certain Python stdlib modules to package them separately: - Debian may require python3-venv to provide "ensurepip" - NetBSD may require py310-expat to provide "pyexpat" * (* Or whichever version is current for NetBSD.) Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: John Snow <jsnow@redhat.com> Message-Id: <20230511035435.734312-4-jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>	3 years ago
John Snow	6832189fd7	python: drop pipenv The pipenv tool was nice in theory, but in practice it's just too hard to update selectively, and it makes using it a pain. The qemu.qmp repo dropped pipenv support a while back and it's been functioning just fine, so I'm backporting that change here to qemu.git. Signed-off-by: John Snow <jsnow@redhat.com> Message-id: 20230210003147.1309376-3-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>	3 years ago
John Snow	37094b6dd5	python: rename qemu.aqmp to qemu.qmp Now that we are fully switched over to the new QMP library, move it back over the old namespace. This is being done primarily so that we may upload this package simply as "qemu.qmp" without introducing confusion over whether or not "aqmp" is a new protocol or not. The trade-off is increased confusion inside the QEMU developer tree. Sorry! Note: the 'private' member "_aqmp" in legacy.py also changes to "_qmp"; not out of necessity, but just to remove any traces of the "aqmp" name. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Beraldo Leal <bleal@redhat.com> Acked-by: Hanna Reitz <hreitz@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@openvz.org> Message-id: 20220330172812.3427355-8-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>	4 years ago
John Snow	481607c7d3	python/aqmp: add start_server() and accept() methods Add start_server() and accept() methods that can be used instead of start_server_and_accept() to allow more fine-grained control over the incoming connection process. (Eagle-eyed reviewers will surely notice that it's a bit weird that "CONNECTING" is a state that's shared between both the start_server() and connect() states. That's absolutely true, and it's very true that checking on the presence of _accepted as an indicator of state is a hack. That's also very certainly true. But ... this keeps client code an awful lot simpler, as it doesn't have to care exactly how the connection is being made, just that it is. Is it worth disrupting that simplicity in order to provide a better state guard on `accept()`? Hm.) Signed-off-by: John Snow <jsnow@redhat.com> Acked-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Message-id: 20220225205948.3693480-9-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>	4 years ago
John Snow	5e9902a030	python/aqmp: refactor _do_accept() into two distinct steps Refactor _do_accept() into _do_start_server() and _do_accept(). As of this commit, the former calls the latter, but in subsequent commits they'll be split apart. (So please forgive the misnomer for _do_start_server(); it will live up to its name shortly, and the docstring will be updated then too. I'm just cutting down on some churn.) Signed-off-by: John Snow <jsnow@redhat.com> Acked-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Message-id: 20220225205948.3693480-7-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>	4 years ago
John Snow	68a6cf3ffe	python/aqmp: remove _new_session and _establish_connection These two methods attempted to entirely envelop the logic of establishing a connection to a peer start to finish. However, we need to break apart the incoming connection step into more granular steps. We will no longer be able to reasonably constrain the logic inside of these helper functions. So, remove them - with _session_guard(), they no longer serve a real purpose. Although the public API doesn't change, the internal API does. Now that there are no intermediary methods between e.g. connect() and _do_connect(), there's no hook where the runstate is set. As a result, the test suite changes a little to cope with the new semantics of _do_accept() and _do_connect(). Lastly, take some pieces of the now-deleted docstrings and move them up to the public interface level. They were a little more detailed, and it won't hurt to keep them. Signed-off-by: John Snow <jsnow@redhat.com> Acked-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Message-id: 20220225205948.3693480-4-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>	4 years ago
John Snow	0ba4e76b23	python/aqmp: rename 'accept()' to 'start_server_and_accept()' Previously, I had a method named "accept()" that under-the-hood calls bind(2), listen(2) and accept(2). I meant this as a simplification and counterpart to the one-shot "connect()" method. This is confusing to readers who expect accept() to mean just accept(2). Since I need to split apart the "accept()" method into multiple methods anyway (one of which strongly resembling accept(2)), it feels pertinent to rename this method now. Rename this all-in-one method "start_server_and_accept()" instead. Signed-off-by: John Snow <jsnow@redhat.com> Acked-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Message-id: 20220225205948.3693480-3-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>	4 years ago
John Snow	762c280d5f	Python: add setuptools v60.0 workaround Setuptools v60 and later include a bundled version of distutils, a deprecated standard library scheduled for removal in future versions of Python. Setuptools v60 is only possible to install for Python 3.7 and later. Python has a distutils.sysconfig.get_python_lib() function that returns '/usr/lib/pythonX.Y' on posix systems. RPM-based systems actually use '/usr/lib64/pythonX.Y' instead, so Fedora patches stdlib distutils for Python 3.7 and Python 3.8 to return the correct value. Python 3.9 and later introduce a sys.platlibdir property, which returns the correct value on RPM-based systems. The change to a distutils package not provided by Fedora on Python 3.7 and 3.8 causes a regression in distutils.sysconfig.get_python_lib() that ultimately causes false positives to be emitted by pylint, because it can no longer find the system source libraries. Many Python tools are fairly aggressive about updating setuptools packages, and so even though this package is a fair bit newer than Python 3.7/3.8, it's not entirely unreasonable for a given user to have such a modern package with a fairly old Python interpreter. Updates to Python 3.7 and Python 3.8 are being produced for Fedora which will fix the problem on up-to-date systems. Until then, we can force the loading of platform-provided distutils when running the pylint test. This is the least-invasive yet most comprehensive fix. References: https://github.com/pypa/setuptools/pull/2896 https://github.com/PyCQA/pylint/issues/5704 https://github.com/pypa/distutils/issues/110 Signed-off-by: John Snow <jsnow@redhat.com> Message-id: 20220204221804.2047468-2-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>	4 years ago
John Snow	461044ceb4	python: Add iotest linters to test suite Run mypy and pylint on the iotests files directly from the Python CI test infrastructure. This ensures that any accidental breakages to the qemu.[qmp\|aqmp\|machine\|utils] packages will be caught by that test suite. It also ensures that these linters are run with well-known versions and test against a wide variety of python versions, which helps to find accidental cross-version python compatibility issues. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Hanna Reitz <hreitz@redhat.com> Message-id: 20211019144918.3159078-15-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>	5 years ago
John Snow	8193b9d148	python/aqmp: add LineProtocol tests Tests a real connect, a real accept, and really sending and receiving a message over a UNIX socket. Brings coverage of protocol.py up to ~93%. Signed-off-by: John Snow <jsnow@redhat.com> Message-id: 20210915162955.333025-27-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>	5 years ago
John Snow	a1f71b61ea	python/aqmp: add AsyncProtocol unit tests This tests most of protocol.py -- From a hacked up Coverage.py run, it's at about 86%. There's a few error cases that aren't very well tested yet, they're hard to induce artificially so far. I'm working on it. Signed-off-by: John Snow <jsnow@redhat.com> Message-id: 20210915162955.333025-26-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>	5 years ago
John Snow	0d52c19a59	python: only check qemu/ subdir with flake8 flake8 is a little eager to check everything it can. Limit it to checking inside the qemu namespace directory only. Update setup.cfg now that the exclude patterns are no longer necessary. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com> Tested-by: Wainer dos Santos Moschetta <wainersm@redhat.com> Message-id: 20210629214323.1329806-11-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>	5 years ago
John Snow	31622b2a8a	python: add avocado-framework and tests Try using avocado to manage our various tests; even though right now they're only invoking shell scripts and not really running any python-native code. Create tests/, and add shell scripts which call out to mypy, flake8, pylint and isort to enforce the standards in this directory. Add avocado-framework to the setup.cfg development dependencies, and add avocado.cfg to store some preferences for how we'd like the test output to look. Finally, add avocado-framework to the Pipfile environment and lock the new dependencies. We are using avocado >= 87.0 here to take advantage of some features that Cleber has helpfully added to make the test output here very friendly and easy to read for developers that might chance upon the output in Gitlab CI. [Note: ALL of the dependencies get updated to the most modern versions that exist at the time of this writing. No way around it that I have seen. Not ideal, but so it goes.] Provided you have the right development dependencies (mypy, flake8, isort, pylint, and now avocado-framework) You should be able to run "avocado --config avocado.cfg run tests/" from the python folder to run all of these linters with the correct arguments. (A forthcoming commit adds the much easier 'make check'.) Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Tested-by: Cleber Rosa <crosa@redhat.com> Message-id: 20210527211715.394144-28-jsnow@redhat.com Signed-off-by: John Snow <jsnow@redhat.com>	5 years ago

18 Commits (27f347e6a1d269c533633c812321cabb249eada8)