riscv/qemu - qemu - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Steve Sistare	a3eae205c6	migration: cpr-exec mode Add the cpr-exec migration mode. Usage: qemu-system-$arch -machine aux-ram-share=on ... migrate_set_parameter mode cpr-exec migrate_set_parameter cpr-exec-command \ <arg1> <arg2> ... -incoming <uri-1> \ migrate -d <uri-1> The migrate command stops the VM, saves state to uri-1, directly exec's a new version of QEMU on the same host, replacing the original process while retaining its PID, and loads state from uri-1. Guest RAM is preserved in place, albeit with new virtual addresses. The new QEMU process is started by exec'ing the command specified by the @cpr-exec-command parameter. The first word of the command is the binary, and the remaining words are its arguments. The command may be a direct invocation of new QEMU, or may be a non-QEMU command that exec's the new QEMU binary. This mode creates a second migration channel that is not visible to the user. At the start of migration, old QEMU saves CPR state to the second channel, and at the end of migration, it tells the main loop to call cpr_exec. New QEMU loads CPR state early, before objects are created. Because old QEMU terminates when new QEMU starts, one cannot stream data between the two, so uri-1 must be a type, such as a file, that accepts all data before old QEMU exits. Otherwise, old QEMU may quietly block writing to the channel. Memory-backend objects must have the share=on attribute, but memory-backend-epc is not supported. The VM must be started with the '-machine aux-ram-share=on' option, which allows anonymous memory to be transferred in place to the new process. The memfds are kept open across exec by clearing the close-on-exec flag, their values are saved in CPR state, and they are mmap'd in new QEMU. Signed-off-by: Steve Sistare <steven.sistare@oracle.com> Acked-by: Markus Armbruster <armbru@redhat.com> Link: https://lore.kernel.org/r/1759332851-370353-7-git-send-email-steven.sistare@oracle.com Signed-off-by: Peter Xu <peterx@redhat.com>	7 months ago
Steve Sistare	f57ff59f1e	migration: cpr-exec-command parameter Create the cpr-exec-command migration parameter, defined as a list of strings. It will be used for cpr-exec migration mode in a subsequent patch, and contains forward references to cpr-exec mode in the qapi doc. No functional change, except that cpr-exec-command is shown by the 'info migrate' command. Signed-off-by: Steve Sistare <steven.sistare@oracle.com> Acked-by: Markus Armbruster <armbru@redhat.com> Link: https://lore.kernel.org/r/1759332851-370353-5-git-send-email-steven.sistare@oracle.com Signed-off-by: Peter Xu <peterx@redhat.com>	7 months ago
Markus Armbruster	d4d91ed42e	qapi: Add more cross-references We recently (merge commit `504632dcc6`) enclosed command and type names in `backquotes`, so they become links in generated HTML. Take care of a few we missed. Signed-off-by: Markus Armbruster <armbru@redhat.com> Message-ID: <20250717115751.3832597-1-armbru@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: John Snow <jsnow@redhat.com>	9 months ago
John Snow	b51ea1c421	qapi: add cross-references to migration.json Note that a reference to MIGRATION needs to be disambiguated with a :qapi:event: prefix. Without this, Sphinx complains more than one target found for 'any' cross-reference 'MIGRATION': could be :std:ref:`Migration framework` or :qapi:event:`QMP:migration.MIGRATION` Signed-off-by: John Snow <jsnow@redhat.com> Message-ID: <20250711054005.60969-9-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> [Commit message amended to explain need for :qapi:event:] Signed-off-by: Markus Armbruster <armbru@redhat.com>	9 months ago
John Snow	0462da9d6b	qapi: remove trivial "Returns:" sections The new qapidoc.py can generate "Returns" statements with type information just fine, so we can remove it from the source where it doesn't add anything particularly novel or helpful and just repeats the type info. This patch is fairly "gentle" and doesn't aggressively touch other "Returns" lines that could be rephrased to omit repeating type information; it only removes lines that appear appropriate to wholly remove. To help facilitate auto-generated placement, a few doc blocks have a "TODO:" line inserted to help the placement algorithm differentiate the introductory paragraph(s) from the rest of the documentation. The auto-generated returns are in the exact same spot, except for query-migrationthreads, query-machines, and x-query-virtio-queue-element. These auto-generation moves to a better spot. Signed-off-by: John Snow <jsnow@redhat.com> Message-ID: <20250711051045.51110-4-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> [Two more Returns: removed, commit message amended to explain auto-generated returns generated into a different spot] Signed-off-by: Markus Armbruster <armbru@redhat.com>	9 months ago
John Snow	6c10778826	docs/sphinx: remove special parsing for freeform sections Remove the QAPI doc section heading syntax, use plain rST section headings instead. Tests and documentation are updated to match. Interestingly, Plain rST headings work fine before this patch, except for over- and underlining with '=', which the doc parser rejected as invalid QAPI doc section heading in free-form comments. Signed-off-by: John Snow <jsnow@redhat.com> Message-ID: <20250618165353.1980365-5-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> [Add more detail to commit message] Signed-off-by: Markus Armbruster <armbru@redhat.com>	10 months ago
Peter Xu	3345fb3b6d	migration/postcopy: Add latency distribution report for blocktime Add the latency distribution too for blocktime, using order-of-two buckets. It accounts for all the faults, from either vCPU or non-vCPU threads. With prior rework, it's very easy to achieve by adding an array to account for faults in each buckets. Sample output for HMP (while for QMP it's simply an array): Postcopy Latency Distribution: [ 1 us - 2 us ]: 0 [ 2 us - 4 us ]: 0 [ 4 us - 8 us ]: 1 [ 8 us - 16 us ]: 2 [ 16 us - 32 us ]: 2 [ 32 us - 64 us ]: 3 [ 64 us - 128 us ]: 10169 [ 128 us - 256 us ]: 50151 [ 256 us - 512 us ]: 12876 [ 512 us - 1 ms ]: 97 [ 1 ms - 2 ms ]: 42 [ 2 ms - 4 ms ]: 44 [ 4 ms - 8 ms ]: 93 [ 8 ms - 16 ms ]: 138 [ 16 ms - 32 ms ]: 0 [ 32 ms - 65 ms ]: 0 [ 65 ms - 131 ms ]: 0 [ 131 ms - 262 ms ]: 0 [ 262 ms - 524 ms ]: 0 [ 524 ms - 1 sec ]: 0 [ 1 sec - 2 sec ]: 0 [ 2 sec - 4 sec ]: 0 [ 4 sec - 8 sec ]: 0 [ 8 sec - 16 sec ]: 0 Cc: Markus Armbruster <armbru@redhat.com> Acked-by: Dr. David Alan Gilbert <dave@treblig.org> Reviewed-by: Fabiano Rosas <farosas@suse.de> Link: https://lore.kernel.org/r/20250613141217.474825-15-peterx@redhat.com Signed-off-by: Peter Xu <peterx@redhat.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>	10 months ago
Peter Xu	ed23a15976	migration/postcopy: blocktime allows track / report non-vCPU faults When used to report page fault latencies, the blocktime feature can be almost useless when KVM async page fault is enabled, because in most cases such remote fault will kickoff async page faults, then it's not trackable from blocktime layer. After all these recent rewrites to blocktime layer, it's finally so easy to also support tracking non-vCPU faults. It'll be even faster if we could always index fault records with TIDs, unfortunately we need to maintain the blocktime API which report things in vCPU indexes. Of course this can work not only for kworkers, but also any guest accesses that may reach a missing page, for example, very likely when in the QEMU main thread too (and all other threads whenever applicable). In this case, we don't care about "how long the threads are blocked", but we only care about "how long the fault will be resolved". Cc: Markus Armbruster <armbru@redhat.com> Cc: Dr. David Alan Gilbert <dave@treblig.org> Reviewed-by: Fabiano Rosas <farosas@suse.de> Tested-by: Mario Casquero <mcasquer@redhat.com> Link: https://lore.kernel.org/r/20250613141217.474825-14-peterx@redhat.com Signed-off-by: Peter Xu <peterx@redhat.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>	10 months ago
Peter Xu	b4c82b4288	migration/postcopy: Report fault latencies in blocktime Blocktime so far only cares about the time one vcpu (or the whole system) got blocked. It would be also be helpful if it can also report the latency of page requests, which could be very sensitive during postcopy. Blocktime itself is sometimes not very important, especially when one thinks about KVM async PF support, which means vCPUs are literally almost not blocked at all because the guest OS is smart enough to switch to another task when a remote fault is needed. However, latency is still sensitive and important because even if the guest vCPU is running on threads that do not need a remote fault, the workload that accesses some missing page is still affected. Add two entries to the report, showing how long it takes to resolve a remote fault. Mention in the QAPI doc that this is not the real average fault latency, but only the ones that was requested for a remote fault. Unwrap get_vcpu_blocktime_list() so we don't need to walk the list twice, meanwhile add the entry checks in qtests for all postcopy tests. Cc: Markus Armbruster <armbru@redhat.com> Cc: Dr. David Alan Gilbert <dave@treblig.org> Reviewed-by: Fabiano Rosas <farosas@suse.de> Tested-by: Mario Casquero <mcasquer@redhat.com> Link: https://lore.kernel.org/r/20250613141217.474825-9-peterx@redhat.com Signed-off-by: Peter Xu <peterx@redhat.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>	10 months ago
Steve Sistare	7437caad20	vfio: doc changes for cpr Update documentation to say that cpr-transfer supports vfio and iommufd. Signed-off-by: Steve Sistare <steven.sistare@oracle.com> Reviewed-by: Cédric Le Goater <clg@redhat.com> Reviewed-by: Fabiano Rosas <farosas@suse.de> Link: https://lore.kernel.org/qemu-devel/1751493538-202042-22-git-send-email-steven.sistare@oracle.com Signed-off-by: Cédric Le Goater <clg@redhat.com>	10 months ago
Markus Armbruster	188b31ad42	qapi: Use proper markup instead of CAPS for emphasis in doc comments Signed-off-by: Markus Armbruster <armbru@redhat.com> Message-ID: <20250527073916.1243024-9-armbru@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com>	11 months ago
Markus Armbruster	6263225492	qapi: Correct spelling of QEMU in doc comments Improve awkward phrasing in migrate-incoming While there. Signed-off-by: Markus Armbruster <armbru@redhat.com> Message-ID: <20250527073916.1243024-7-armbru@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com>	11 months ago
Markus Armbruster	c1a6aa1d44	qapi: Avoid breaking lines within (since X.Y) Easier on the eyes and for grep. Signed-off-by: Markus Armbruster <armbru@redhat.com> Message-ID: <20250527073916.1243024-5-armbru@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com>	11 months ago
Markus Armbruster	69d68fb34e	qapi: Tidy up whitespace in doc comments Signed-off-by: Markus Armbruster <armbru@redhat.com> Message-ID: <20250527073916.1243024-3-armbru@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com>	11 months ago
Markus Armbruster	5e203f73c2	qapi: Tidy up run-together sentences in doc comments Fixes: `a937b6aa73` (qapi: Reformat doc comments to conform to current conventions) Signed-off-by: Markus Armbruster <armbru@redhat.com> Message-ID: <20250527073916.1243024-2-armbru@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com>	11 months ago
Pierrick Bouvier	e1a80c3241	qapi: use imperative style in documentation As requested by Markus: > We prefer imperative mood "Return" over "Returns". Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org> Message-ID: <20250522190542.588267-14-pierrick.bouvier@linaro.org> Reviewed-by: Markus Armbruster <armbru@redhat.com> [Change several more]	11 months ago
Markus Armbruster	c2fb6eaeb9	qapi/migration: Deprecate migrate argument @detach Argument @detach has always been ignored. Start the clock to get rid of it. Cc: Peter Xu <peterx@redhat.com> Cc: Fabiano Rosas <farosas@suse.de> Signed-off-by: Markus Armbruster <armbru@redhat.com> Message-ID: <20250521063711.29840-3-armbru@redhat.com> ACKed-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Fabiano Rosas <farosas@suse.de> Reviewed-by: Peter Xu <peterx@redhat.com>	11 months ago
Fabiano Rosas	24f4c80cfc	migration: Update migrate_cancel documentation Update the migrate_cancel command documentation with a few words about postcopy and the expected state of the machine after migration. Acked-by: Markus Armbruster <armbru@redhat.com> Reviewed-by: Peter Xu <peterx@redhat.com> Message-ID: <20250213175927.19642-10-farosas@suse.de> Signed-off-by: Fabiano Rosas <farosas@suse.de>	1 year ago
Peter Xu	4881411136	migration: Always set DEVICE state DEVICE state was introduced back in 2017: https://lore.kernel.org/qemu-devel/20171020090556.18631-1-dgilbert@redhat.com/ Quote from Dave's cover letter, when the pre-switchover phase was enabled, the state transition looks like this: The precopy flow is: active->pre-switchover->device->completed The postcopy flow is: active->pre-switchover->postcopy-active->completed To supplement above, when the cap is not enabled: The precopy flow is: active->completed The postcopy flow is: active->postcopy-active->completed It works for us, though we have some code just to special case these state transitions, so the DEVICE state currently is special only to precopy, and only conditionally. I had a quick discussion with Libvirt developers, it turns out that this may not be necessary. IOW, it seems okay we can have DEVICE state to be generic, so that we don't have over-complicated state machines. It not only helps align all the migration state machine, help cleanup the code path especially on pre-switchover handling (see the patch itself), another side benefit is we can unconditionally have a specific state to mark the switchover phase, which might be helpful for debugging too. This patch makes the DEVICE state to be present always, marking that source QEMU is switching over. Then the state machine will be always as simple as: active-> [pre-switchover->] -> device -> [postcopy-active->] -> complete After the change, no matter whether pre-switchover or postcopy is enabled or not, we always have DEVICE state showing the switchover phase. When pre-switchover enabled, we'll have an extra stage before that. When postcopy is enabled, we'll have an extra stage after that. A few qtests need touch up in QEMU tree for this change: - A few iotest outputs (194, 203, 234, 262, 280) - Teach libqos's migrate() on "device" state Cc: Jiri Denemark <jdenemar@redhat.com> Cc: Daniel P. Berrangé <berrange@redhat.com> Cc: Dr. David Alan Gilbert <dave@treblig.org> Signed-off-by: Peter Xu <peterx@redhat.com> Tested-by: Jiri Denemark <jdenemar@redhat.com> Reviewed-by: Juraj Marcin <jmarcin@redhat.com> Link: https://lore.kernel.org/r/20250114230746.3268797-15-peterx@redhat.com Signed-off-by: Fabiano Rosas <farosas@suse.de>	1 year ago
Steve Sistare	624e6e654e	migration: cpr-transfer mode Add the cpr-transfer migration mode, which allows the user to transfer a guest to a new QEMU instance on the same host with minimal guest pause time, by preserving guest RAM in place, albeit with new virtual addresses in new QEMU, and by preserving device file descriptors. Pages that were locked in memory for DMA in old QEMU remain locked in new QEMU, because the descriptor of the device that locked them remains open. cpr-transfer preserves memory and devices descriptors by sending them to new QEMU over a unix domain socket using SCM_RIGHTS. Such CPR state cannot be sent over the normal migration channel, because devices and backends are created prior to reading the channel, so this mode sends CPR state over a second "cpr" migration channel. New QEMU reads the cpr channel prior to creating devices or backends. The user specifies the cpr channel in the channel arguments on the outgoing side, and in a second -incoming command-line parameter on the incoming side. The user must start old QEMU with the the '-machine aux-ram-share=on' option, which allows anonymous memory to be transferred in place to the new process by transferring a memory descriptor for each ram block. Memory-backend objects must have the share=on attribute, but memory-backend-epc is not supported. The user starts new QEMU on the same host as old QEMU, with command-line arguments to create the same machine, plus the -incoming option for the main migration channel, like normal live migration. In addition, the user adds a second -incoming option with channel type "cpr". This CPR channel must support file descriptor transfer with SCM_RIGHTS, i.e. it must be a UNIX domain socket. To initiate CPR, the user issues a migrate command to old QEMU, adding a second migration channel of type "cpr" in the channels argument. Old QEMU stops the VM, saves state to the migration channels, and enters the postmigrate state. New QEMU mmap's memory descriptors, and execution resumes. The implementation splits qmp_migrate into start and finish functions. Start sends CPR state to new QEMU, which responds by closing the CPR channel. Old QEMU detects the HUP then calls finish, which connects the main migration channel. In summary, the usage is: qemu-system-$arch -machine aux-ram-share=on ... start new QEMU with "-incoming <main-uri> -incoming <cpr-channel>" Issue commands to old QEMU: migrate_set_parameter mode cpr-transfer {"execute": "migrate", ... {"channel-type": "main"...}, {"channel-type": "cpr"...} ... } Signed-off-by: Steve Sistare <steven.sistare@oracle.com> Reviewed-by: Peter Xu <peterx@redhat.com> Acked-by: Markus Armbruster <armbru@redhat.com> Link: https://lore.kernel.org/r/1736967650-129648-17-git-send-email-steven.sistare@oracle.com Signed-off-by: Fabiano Rosas <farosas@suse.de>	1 year ago
Peter Xu	228529d1fe	migration: Deprecate query-migrationthreads command Per previous discussion [1,2], this patch deprecates query-migrationthreads command. To summarize, the major reason of the deprecation is due to no sensible way to consume the API properly: (1) The reported list of threads are incomplete (ignoring destination threads and non-multifd threads). (2) For CPU pinning, there's no way to properly pin the threads with the API if the threads will start running right away after migration threads can be queried, so the threads will always run on the default cores for a short window. (3) For VM debugging, one can use "-name $VM,debug-threads=on" instead, which will provide proper names for all migration threads. [1] https://lore.kernel.org/r/20240930195837.825728-1-peterx@redhat.com [2] https://lore.kernel.org/r/20241011153417.516715-1-peterx@redhat.com Reviewed-by: Fabiano Rosas <farosas@suse.de> Acked-by: Markus Armbruster <armbru@redhat.com> Link: https://lore.kernel.org/r/20241022194501.1022443-1-peterx@redhat.com Signed-off-by: Peter Xu <peterx@redhat.com>	2 years ago
Fabiano Rosas	73581a041e	migration: Deprecate zero-blocks capability The zero-blocks capability was meant to be used along with the block migration, which has been removed already in commit `eef0bae3a7` ("migration: Remove block migration"). Setting zero-blocks is currently a noop, but the outright removal of the capability would cause and error in case some users are still setting it. Put the capability through the deprecation process. Signed-off-by: Fabiano Rosas <farosas@suse.de> Reviewed-by: Peter Xu <peterx@redhat.com> Link: https://lore.kernel.org/r/20240919134626.166183-4-dave@treblig.org Signed-off-by: Peter Xu <peterx@redhat.com>	2 years ago
Markus Armbruster	7b29353fdd	qapi: Smarter camel_to_upper() to reduce need for 'prefix' camel_to_upper() converts its argument from camel case to upper case with '_' between words. Used for generated enumeration constant prefixes. When some of the words are spelled all caps, where exactly to insert '_' is guesswork. camel_to_upper()'s guesses are bad enough in places to make people override them with a 'prefix' in the schema. Rewrite it to guess better: 1. Insert '_' after a non-upper case character followed by an upper case character: OneTwo -> ONE_TWO One2Three -> ONE2_THREE 2. Insert '_' before the last upper case character followed by a non-upper case character: ACRONYMWord -> ACRONYM_Word Except at the beginning (as in OneTwo above), or when there is already one: AbCd -> AB_CD This changes the default enumeration constant prefix for a number of enums. Generated enumeration constants change only where the default is not overridden with 'prefix'. The following enumerations without a 'prefix' change: enum old camel_to_upper() new camel_to_upper() ------------------------------------------------------------------ DisplayGLMode DISPLAYGL_MODE DISPLAY_GL_MODE EbpfProgramID EBPF_PROGRAMID EBPF_PROGRAM_ID HmatLBDataType HMATLB_DATA_TYPE HMAT_LB_DATA_TYPE HmatLBMemoryHierarchy HMATLB_MEMORY_HIERARCHY HMAT_LB_MEMORY_HIERARCHY MultiFDCompression MULTIFD_COMPRESSION MULTI_FD_COMPRESSION OffAutoPCIBAR OFF_AUTOPCIBAR OFF_AUTO_PCIBAR QCryptoBlockFormat Q_CRYPTO_BLOCK_FORMAT QCRYPTO_BLOCK_FORMAT QCryptoBlockLUKSKeyslotState Q_CRYPTO_BLOCKLUKS_KEYSLOT_STATE QCRYPTO_BLOCK_LUKS_KEYSLOT_STATE QKeyCode Q_KEY_CODE QKEY_CODE XDbgBlockGraphNodeType X_DBG_BLOCK_GRAPH_NODE_TYPE XDBG_BLOCK_GRAPH_NODE_TYPE TestUnionEnumA TEST_UNION_ENUMA TEST_UNION_ENUM_A Add a 'prefix' so generated code doesn't change now. Subsequent commits will remove most of them again. Two will remain: MULTIFD_COMPRESSION, because migration code generally spells "multifd" that way, and Q_KEY_CODE, because that one is baked into subprojects/keycodemapdb/tools/keymap-gen. The following enumerations with a 'prefix' change so that the prefix is now superfluous: enum old camel_to_upper() new camel_to_upper() [equal to prefix] ------------------------------------------------------------------ BlkdebugIOType BLKDEBUGIO_TYPE BLKDEBUG_IO_TYPE QCryptoTLSCredsEndpoint Q_CRYPTOTLS_CREDS_ENDPOINT QCRYPTO_TLS_CREDS_ENDPOINT QCryptoSecretFormat Q_CRYPTO_SECRET_FORMAT QCRYPTO_SECRET_FORMAT QCryptoCipherMode Q_CRYPTO_CIPHER_MODE QCRYPTO_CIPHER_MODE QCryptodevBackendType Q_CRYPTODEV_BACKEND_TYPE QCRYPTODEV_BACKEND_TYPE QType [builtin] Q_TYPE QTYPE Drop these prefixes. The following enumerations with a 'prefix' change without making the 'prefix' superfluous: enum old camel_to_upper() new camel_to_upper() [equal to prefix] prefix ------------------------------------------------------------------ CpuS390Entitlement CPUS390_ENTITLEMENT CPU_S390_ENTITLEMENT S390_CPU_ENTITLEMENT CpuS390Polarization CPUS390_POLARIZATION CPU_S390_POLARIZATION S390_CPU_POLARIZATION CpuS390State CPUS390_STATE CPU_S390_STATE S390_CPU_STATE QAuthZListFormat Q_AUTHZ_LIST_FORMAT QAUTH_Z_LIST_FORMAT QAUTHZ_LIST_FORMAT QAuthZListPolicy Q_AUTHZ_LIST_POLICY QAUTH_Z_LIST_POLICY QAUTHZ_LIST_POLICY QCryptoAkCipherAlgorithm Q_CRYPTO_AK_CIPHER_ALGORITHM QCRYPTO_AK_CIPHER_ALGORITHM QCRYPTO_AKCIPHER_ALG QCryptoAkCipherKeyType Q_CRYPTO_AK_CIPHER_KEY_TYPE QCRYPTO_AK_CIPHER_KEY_TYPE QCRYPTO_AKCIPHER_KEY_TYPE QCryptoCipherAlgorithm Q_CRYPTO_CIPHER_ALGORITHM QCRYPTO_CIPHER_ALGORITHM QCRYPTO_CIPHER_ALG QCryptoHashAlgorithm Q_CRYPTO_HASH_ALGORITHM QCRYPTO_HASH_ALGORITHM QCRYPTO_HASH_ALG QCryptoIVGenAlgorithm Q_CRYPTOIV_GEN_ALGORITHM QCRYPTO_IV_GEN_ALGORITHM QCRYPTO_IVGEN_ALG QCryptoRSAPaddingAlgorithm Q_CRYPTORSA_PADDING_ALGORITHM QCRYPTO_RSA_PADDING_ALGORITHM QCRYPTO_RSA_PADDING_ALG QCryptodevBackendAlgType Q_CRYPTODEV_BACKEND_ALG_TYPE QCRYPTODEV_BACKEND_ALG_TYPE QCRYPTODEV_BACKEND_ALG QCryptodevBackendServiceType Q_CRYPTODEV_BACKEND_SERVICE_TYPE QCRYPTODEV_BACKEND_SERVICE_TYPE QCRYPTODEV_BACKEND_SERVICE Subsequent commits will tweak things to remove most of these prefixes. Only QAUTHZ_LIST_FORMAT and QAUTHZ_LIST_POLICY will remain. Signed-off-by: Markus Armbruster <armbru@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Message-ID: <20240904111836.3273842-2-armbru@redhat.com>	2 years ago
Bryan Zhang	80484f9459	migration: Introduce 'qatzip' compression method Adds support for 'qatzip' as an option for the multifd compression method parameter, and implements using QAT for 'qatzip' compression and decompression. Acked-by: Markus Armbruster <armbru@redhat.com> Reviewed-by: Fabiano Rosas <farosas@suse.de> Reviewed-by: Prasad Pandit <pjp@fedoraproject.org> Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com> Signed-off-by: Hao Xiang <hao.xiang@linux.dev> Signed-off-by: Yichen Wang <yichen.wang@bytedance.com> Link: https://lore.kernel.org/r/20240830232722.58272-5-yichen.wang@bytedance.com Signed-off-by: Peter Xu <peterx@redhat.com>	2 years ago
Bryan Zhang	86c6eb1f39	migration: Add migration parameters for QATzip Adds support for migration parameters to control QATzip compression level. Acked-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com> Signed-off-by: Hao Xiang <hao.xiang@linux.dev> Signed-off-by: Yichen Wang <yichen.wang@bytedance.com> Reviewed-by: Fabiano Rosas <farosas@suse.de> Reviewed-by: Prasad Pandit <pjp@fedoraproject.org> Link: https://lore.kernel.org/r/20240830232722.58272-4-yichen.wang@bytedance.com Signed-off-by: Peter Xu <peterx@redhat.com>	2 years ago
Markus Armbruster	01bed0ff14	qapi: Refill doc comments to conform to conventions Sweep the entire documentation again. Last done in commit `209e64d9ed` (qapi: Refill doc comments to conform to current conventions). To check the generated documentation does not change, I compared the generated HTML before and after this commit with "wdiff -3". Finds no differences. Comparing with diff is not useful, as the reflown paragraphs are visible there. Signed-off-by: Markus Armbruster <armbru@redhat.com> Message-ID: <20240729065220.860163-1-armbru@redhat.com> [Straightforward conflict with commit `442110bc6f` resolved]	2 years ago
John Snow	6f07c59fb1	qapi: convert "Example" sections with longer prose These examples require longer explanations or have explanations that require markup to look reasonable when rendered and so use the longer form of the ".. qmp-example::" directive. By using the :annotated: option, the content in the example block is assumed not to be a code block literal and is instead parsed as normal rST - with the exception that any code literal blocks after `::` will assumed to be a QMP code literal block. Note: There's one title-less conversion in this patch that comes along for the ride because it's part of a larger "Examples" block that was better to convert all at once. See commit-5: "docs/qapidoc: create qmp-example directive", for a detailed explanation of this custom directive syntax. See commit+1: "qapi: remove "Example" doc section" for a detailed explanation of why. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Message-ID: <20240717021312.606116-9-jsnow@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>	2 years ago
John Snow	a9eab6e2e6	qapi: convert "Example" sections with titles When an Example section has a brief explanation, convert it to a qmp-example:: section using the :title: option. Rule of thumb: If the title can fit on a single line and requires no rST markup, it's a good candidate for using the :title: option of qmp-example. In this patch, trailing punctuation is removed from the title section for consistent headline aesthetics. In just one case, specifics of the example are removed to make the title read better. See commit-4: "docs/qapidoc: create qmp-example directive", for a detailed explanation of this custom directive syntax. See commit+2: "qapi: remove "Example" doc section" for a detailed explanation of why. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Message-ID: <20240717021312.606116-8-jsnow@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>	2 years ago
John Snow	14b48aaab9	qapi: convert "Example" sections without titles Use the no-option form of ".. qmp-example::" to convert any Examples that do not have any form of caption or explanation whatsoever. Note that in a few cases, example sections are split into two or more separate example blocks. This is only done stylistically to create a delineation between two or more logically independent examples. See commit-3: "docs/qapidoc: create qmp-example directive", for a detailed explanation of this custom directive syntax. See commit+3: "qapi: remove "Example" doc section" for a detailed explanation of why. Note: an empty "TODO" line was added to announce-self to keep the example from floating up into the body; this will be addressed more rigorously in the new qapidoc generator. Signed-off-by: John Snow <jsnow@redhat.com> Message-ID: <20240717021312.606116-7-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> [Markup fixed in one place] Signed-off-by: Markus Armbruster <armbru@redhat.com>	2 years ago
John Snow	d461c27973	qapi: convert "Note" sections to plain rST We do not need a dedicated section for notes. By eliminating a specially parsed section, these notes can be treated as normal rST paragraphs in the new QMP reference manual, and can be placed and styled much more flexibly. Convert all existing "Note" and "Notes" sections to pure rST. As part of the conversion, capitalize the first letter of each sentence and add trailing punctuation where appropriate to ensure notes look sensible and consistent in rendered HTML documentation. Markup is also re-aligned to the de-facto standard of 3 spaces for directives. Update docs/devel/qapi-code-gen.rst to reflect the new paradigm, and update the QAPI parser to prohibit "Note" sections while suggesting a new syntax. The exact formatting to use is a matter of taste, but a good candidate is simply: .. note:: lorem ipsum ... ... dolor sit amet ... ... consectetur adipiscing elit ... ... but there are other choices, too. The Sphinx readthedocs theme offers theming for the following forms (capitalization unimportant); all are adorned with a (!) symbol () in the title bar for rendered HTML docs. See https://sphinx-rtd-theme.readthedocs.io/en/stable/demo/demo.html#admonitions for examples of each directive/admonition in use. These are rendered in orange: .. Attention:: ... .. Caution:: ... .. WARNING:: ... These are rendered in red: .. DANGER:: ... .. Error:: ... These are rendered in green: .. Hint:: ... .. Important:: ... .. Tip:: ... These are rendered in blue: .. Note:: ... .. admonition:: custom title admonition body text This patch uses ".. note::" almost everywhere, with just two "caution" directives. Several instances of "Notes:" have been converted to merely ".. note::", or multiple ".. note::" where appropriate. ".. admonition:: notes" is used in a few places where we had an ordered list of multiple notes that would not make sense as standalone/separate admonitions. Two "Note:" following "Example:" have been turned into ordinary paragraphs within the example. NOTE: Because qapidoc.py does not attempt to preserve source ordering of sections, the conversion of Notes from a "tagged section" to an "untagged section" means that rendering order for some notes may change as a result of this patch. The forthcoming qapidoc.py rewrite strictly preserves source ordering in the rendered documentation, so this issue will be rectified in the new generator. Signed-off-by: John Snow <jsnow@redhat.com> Acked-by: Stefan Hajnoczi <stefanha@redhat.com> [for block*.json] Message-ID: <20240626222128.406106-11-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> [Commit message clarified slightly, period added to one more note] Signed-off-by: Markus Armbruster <armbru@redhat.com>	2 years ago
John Snow	9f2b848857	qapi: fix non-compliant JSON examples The new QMP documentation generator wants to parse all examples as "QMP". We have an existing QMP lexer in docs/sphinx/qmp_lexer.py (Seen in-use here: https://qemu-project.gitlab.io/qemu/interop/bitmaps.html) that allows the use of "->", "<-" and "..." tokens to denote QMP protocol flow with elisions, but otherwise defers to the JSON lexer. To utilize this lexer for the existing QAPI documentation, we need them to conform to a standard so that they lex and render correctly. Once the QMP lexer is active for examples, errant QMP/JSON will produce warning messages and fail the build. Fix any invalid JSON found in QAPI documentation (identified by attempting to lex all examples as QMP; see subsequent commits). Additionally, elisions must be standardized for the QMP lexer; they must be represented as the value "...", so three examples have been adjusted to support that format here. Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Message-ID: <20240626222128.406106-9-jsnow@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>	2 years ago
Peter Xu	4146b77ec7	migration/postcopy: Add postcopy-recover-setup phase This patch adds a migration state on src called "postcopy-recover-setup". The new state will describe the intermediate step starting from when the src QEMU received a postcopy recovery request, until the migration channels are properly established, but before the recovery process take place. The request came from Libvirt where Libvirt currently rely on the migration state events to detect migration state changes. That works for most of the migration process but except postcopy recovery failures at the beginning. Currently postcopy recovery only has two major states: - postcopy-paused: this is the state that both sides of QEMU will be in for a long time as long as the migration channel was interrupted. - postcopy-recover: this is the state where both sides of QEMU handshake with each other, preparing for a continuation of postcopy which used to be interrupted. The issue here is when the recovery port is invalid, the src QEMU will take the URI/channels, noticing the ports are not valid, and it'll silently keep in the postcopy-paused state, with no event sent to Libvirt. In this case, the only thing Libvirt can do is to poll the migration status with a proper interval, however that's less optimal. Considering that this is the only case where Libvirt won't get a notification from QEMU on such events, let's add postcopy-recover-setup state to mimic what we have with the "setup" state of a newly initialized migration, describing the phase of connection establishment. With that, postcopy recovery will have two paths to go now, and either path will guarantee an event generated. Now the events will look like this during a recovery process on src QEMU: - Initially when the recovery is initiated on src, QEMU will go from "postcopy-paused" -> "postcopy-recover-setup". Old QEMUs don't have this event. - Depending on whether the channel re-establishment is succeeded: - In succeeded case, src QEMU will move from "postcopy-recover-setup" to "postcopy-recover". Old QEMUs also have this event. - In failure case, src QEMU will move from "postcopy-recover-setup" to "postcopy-paused" again. Old QEMUs don't have this event. This guarantees that Libvirt will always receive a notification for recovery process properly. One thing to mention is, such new status is only needed on src QEMU not both. On dest QEMU, the state machine doesn't change. Hence the events don't change either. It's done like so because dest QEMU may not have an explicit point of setup start. E.g., it can happen that when dest QEMUs doesn't use migrate-recover command to use a new URI/channel, but the old URI/channels can be reused in recovery, in which case the old ports simply can work again after the network routes are fixed up. Add a new helper postcopy_is_paused() detecting whether postcopy is still paused, taking RECOVER_SETUP into account too. When using it on both src/dst, a slight change is done altogether to always wait for the semaphore before checking the status, because for both sides a sem_post() will be required for a recovery. Cc: Jiri Denemark <jdenemar@redhat.com> Cc: Prasad Pandit <ppandit@redhat.com> Reviewed-by: Fabiano Rosas <farosas@suse.de> Buglink: https://issues.redhat.com/browse/RHEL-38485 Signed-off-by: Peter Xu <peterx@redhat.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>	2 years ago
Fabiano Rosas	b43b61d5be	migration: Add direct-io parameter Add the direct-io migration parameter that tells the migration code to use O_DIRECT when opening the migration stream file whenever possible. This is currently only used with the mapped-ram migration that has a clear window guaranteed to perform aligned writes. Acked-by: Markus Armbruster <armbru@redhat.com> Reviewed-by: Peter Xu <peterx@redhat.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>	2 years ago
Shameer Kolothum	f3d8bb759d	migration/multifd: add uadk compression framework Adds the skeleton to support uadk compression method. Complete functionality will be added in subsequent patches. Acked-by: Markus Armbruster <armbru@redhat.com> Reviewed-by: Fabiano Rosas <farosas@suse.de> Signed-off-by: Shameer Kolothum <shameerali.kolothum.thodi@huawei.com> Reviewed-by: Zhangfei Gao <zhangfei.gao@linaro.org> Signed-off-by: Fabiano Rosas <farosas@suse.de>	2 years ago
Yuan Liu	354cac2859	migration/multifd: add qpl compression method add the Query Processing Library (QPL) compression method Introduce the qpl as a new multifd migration compression method, it can use In-Memory Analytics Accelerator(IAA) to accelerate compression and decompression, which can not only reduce network bandwidth requirement but also reduce host compression and decompression CPU overhead. How to enable qpl compression during migration: migrate_set_parameter multifd-compression qpl There is no qpl compression level parameter added since it only supports level one, users do not need to specify the qpl compression level. Signed-off-by: Yuan Liu <yuan1.liu@intel.com> Reviewed-by: Nanhai Zou <nanhai.zou@intel.com> Reviewed-by: Peter Xu <peterx@redhat.com> Reviewed-by: Fabiano Rosas <farosas@suse.de> [fixed docs spacing in migration.json] Signed-off-by: Fabiano Rosas <farosas@suse.de>	2 years ago
Fabiano Rosas	0222111a22	migration: Remove non-multifd compression The 'compress' migration capability enables the old compression code which has shown issues over the years and is thought to be less stable and tested than the more recent multifd-based compression. The old compression code has been deprecated in 8.2 and now is time to remove it. Deprecation commit `864128df46` ("migration: Deprecate old compression method"). Acked-by: Markus Armbruster <armbru@redhat.com> Reviewed-by: Peter Xu <peterx@redhat.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>	2 years ago
Fabiano Rosas	eef0bae3a7	migration: Remove block migration The block migration has been considered obsolete since QEMU 8.2 in favor of the more flexible storage migration provided by the blockdev-mirror driver. Two releases have passed so now it's time to remove it. Deprecation commit `66db46ca83` ("migration: Deprecate block migration"). Reviewed-by: Markus Armbruster <armbru@redhat.com> Reviewed-by: Peter Xu <peterx@redhat.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>	2 years ago
Fabiano Rosas	18d154f575	migration: Remove 'blk/-b' option from migrate commands The block migration is considered obsolete and has been deprecated in 8.2. Remove the migrate command option that enables it. This only affects the QMP and HMP commands, the feature can still be accessed by setting the migration 'block' capability. The whole feature will be removed in a future patch. Deprecation commit `8846b5bfca` ("migration: migrate 'blk' command option is deprecated."). Reviewed-by: Markus Armbruster <armbru@redhat.com> Reviewed-by: Peter Xu <peterx@redhat.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>	2 years ago
Fabiano Rosas	61c4e39f73	migration: Remove 'inc' option from migrate command The block incremental option for block migration has been deprecated in 8.2 in favor of using the block-mirror feature. Remove it now. Deprecation commit `40101f320d` ("migration: migrate 'inc' command option is deprecated."). Reviewed-by: Markus Armbruster <armbru@redhat.com> Reviewed-by: Peter Xu <peterx@redhat.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>	2 years ago
Fabiano Rosas	f7b1cd3c2e	migration: Remove 'skipped' field from MigrationStats The 'skipped' field of the MigrationStats struct has been deprecated in 8.1. Time to remove it. Deprecation commit `7b24d32634` ("migration: skipped field is really obsolete."). Reviewed-by: Markus Armbruster <armbru@redhat.com> Reviewed-by: Peter Xu <peterx@redhat.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>	2 years ago
Vladimir Sementsov-Ogievskiy	dbea1c89da	qapi: introduce exit-on-error parameter for migrate-incoming Now we do set MIGRATION_FAILED state, but don't give a chance to orchestrator to query migration state and get the error. Let's provide a possibility for QMP-based orchestrators to get an error like with outgoing migration. For hmp_migrate_incoming(), let's enable the new behavior: HMP is not and ABI, it's mostly intended to use by developer and it makes sense not to stop the process. For x-exit-preconfig, let's keep the old behavior: - it's called from init(), so here we want to keep current behavior by default - it does exit on error by itself as well So, if we want to change the behavior of x-exit-preconfig, it should be another patch. Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru> Acked-by: Markus Armbruster <armbru@redhat.com> Reviewed-by: Fabiano Rosas <farosas@suse.de> Reviewed-by: Peter Xu <peterx@redhat.com> Signed-off-by: Fabiano Rosas <farosas@suse.de>	2 years ago
Markus Armbruster	5305a4eeb8	qapi: Correct documentation indentation and whitespace Signed-off-by: Markus Armbruster <armbru@redhat.com> Message-ID: <20240322140910.328840-12-armbru@redhat.com> [Add a previous patch's stray hunk]	2 years ago
Markus Armbruster	209e64d9ed	qapi: Refill doc comments to conform to current conventions For legibility, wrap text paragraphs so every line is at most 70 characters long. To check the generated documentation does not change, I compared the generated HTML before and after this commit with "wdiff -3". Finds no differences. Comparing with diff is not useful, as the refilled paragraphs are visible there. Signed-off-by: Markus Armbruster <armbru@redhat.com> Message-ID: <20240322140910.328840-11-armbru@redhat.com>	2 years ago
Markus Armbruster	73c67f3851	qapi: Start sentences with a capital letter, end them with a period Signed-off-by: Markus Armbruster <armbru@redhat.com> Message-ID: <20240322140910.328840-9-armbru@redhat.com>	2 years ago
Markus Armbruster	7d08424cf7	qapi: Fix abbreviation punctuation in doc comments Signed-off-by: Markus Armbruster <armbru@redhat.com> Message-ID: <20240322140910.328840-8-armbru@redhat.com>	2 years ago
Markus Armbruster	e6c60bf02d	qapi: Fix bogus documentation of query-migrationthreads The doc comment documents an argument that doesn't exist. Would fail compilation if it was marked up correctly. Delete. The Returns: section fails to refer to the data type, leaving the user to guess. Fix that. The command name violates QAPI naming rules: it should be query-migration-threads. Too late to fix. Reported-by: John Snow <jsnow@redhat.com> Fixes: `671326201d` (migration: Introduce interface query-migrationthreads) Signed-off-by: Markus Armbruster <armbru@redhat.com> Message-ID: <20240322135117.195489-4-armbru@redhat.com> Reviewed-by: Fabiano Rosas <farosas@suse.de> Reviewed-by: Peter Xu <peterx@redhat.com> Reviewed-by: John Snow <jsnow@redhat.com>	2 years ago
Markus Armbruster	8eb0a257c5	qapi: Resync MigrationParameter and MigrateSetParameters Enum MigrationParameter mirrors the members of struct MigrateSetParameters. Differences to MigrateSetParameters's member documentation are pointless. Clean them up: * @compress-level, @compress-threads, @decompress-threads, and x-checkpoint-delay are more thoroughly documented for MigrationParameter, so use that version for both. * @max-cpu-throttle is almost the same. Use MigrationParameter's version for both. Signed-off-by: Markus Armbruster <armbru@redhat.com> Message-ID: <20240322135117.195489-3-armbru@redhat.com> Reviewed-by: Fabiano Rosas <farosas@suse.de> Reviewed-by: Peter Xu <peterx@redhat.com>	2 years ago
Markus Armbruster	e8c5503a5c	qapi: Improve migration TLS documentation MigrateSetParameters is about setting parameters, and MigrationParameters is about querying them. Their documentation of @tls-creds and @tls-hostname has residual damage from a failed attempt at de-duplicating them (see commit `de63ab6124` "migrate: Share common MigrationParameters struct" and commit `1bda8b3c69` "migration: Unshare MigrationParameters struct for now"). MigrateSetParameters documentation issues: * It claims plain text mode "was reported by omitting tls-creds" before 2.9. MigrateSetParameters is not used for reporting, so this is misleading. Delete. * It similarly claims hostname defaulting to migration URI "was reported by omitting tls-hostname" before 2.9. Delete as well. Rephrase the remaining @tls-hostname contents for clarity. Enum MigrationParameter mirrors the members of struct MigrateSetParameters. Differences to MigrateSetParameters's member documentation are pointless. Copy the new text to MigrationParameter. MigrationParameters documentation issues: * @tls-creds runs the two last sentences together without punctuation. Fix that. * Much of the contents on @tls-hostname only applies to setting parameters, resulting in confusion. Replace by a suitable abridged version of the new MigrateSetParameters text, and a note on @tls-hostname omission in 2.8. Additional damage is due to flawed doc fix commit `66fcb9d651` (qapi/migration: Add missing tls-authz documentation): since it copied the missing MigrateSetParameters text from MigrationParameters instead of MigrationParameter, the part on recreating @tls-authz on the fly is missing. Copy that, too. Signed-off-by: Markus Armbruster <armbru@redhat.com> Message-ID: <20240322135117.195489-2-armbru@redhat.com> Reviewed-by: Peter Xu <peterx@redhat.com> [Some typos corrected]	2 years ago
Hao Xiang	70c25c92e6	migration/multifd: Enable multifd zero page checking by default. 1. Set default "zero-page-detection" option to "multifd". Now zero page checking can be done in the multifd threads and this becomes the default configuration. 2. Handle migration QEMU9.0 -> QEMU8.2 compatibility. We provide backward compatibility where zero page checking is done from the migration main thread. Signed-off-by: Hao Xiang <hao.xiang@bytedance.com> Reviewed-by: Fabiano Rosas <farosas@suse.de> Reviewed-by: Peter Xu <peterx@redhat.com> Link: https://lore.kernel.org/r/20240311180015.3359271-7-hao.xiang@linux.dev Signed-off-by: Peter Xu <peterx@redhat.com>	2 years ago
Hao Xiang	303e6f54f9	migration/multifd: Implement zero page transmission on the multifd thread. 1. Add zero_pages field in MultiFDPacket_t. 2. Implements the zero page detection and handling on the multifd threads for non-compression, zlib and zstd compression backends. 3. Added a new value 'multifd' in ZeroPageDetection enumeration. 4. Adds zero page counters and updates multifd send/receive tracing format to track the newly added counters. Signed-off-by: Hao Xiang <hao.xiang@bytedance.com> Acked-by: Markus Armbruster <armbru@redhat.com> Reviewed-by: Fabiano Rosas <farosas@suse.de> Link: https://lore.kernel.org/r/20240311180015.3359271-5-hao.xiang@linux.dev Signed-off-by: Peter Xu <peterx@redhat.com>	2 years ago

1 2 3 4 5

211 Commits (e9d02b59316fcbc9650d747b06e2755374ab3b70)