Remove the QAPI doc section heading syntax, use plain rST section
headings instead.
Tests and documentation are updated to match.
Interestingly, Plain rST headings work fine before this patch, except
for over- and underlining with '=', which the doc parser rejected as
invalid QAPI doc section heading in free-form comments.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-ID: <20250618165353.1980365-5-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Add more detail to commit message]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Parser and doc generator cooperate on generating stub documentation for
undocumented members. The parser makes up an ArgSection with an empty
description, and the doc generator makes up a description.
Right now, the made-up ArgSections go into doc.args. However, the new
doc generator uses .all_sections, not .args. So put them into
.all_sections, too.
Insert them right after existing 'member' sections. If there are none,
insert directly after the leading section.
Doesn't affect the old generator, because that one doesn't use
.all_sections.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-ID: <20250311034303.75779-60-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Commit message rewritten]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Makes debugging far more pleasant when you can just print(section) and
get something reasonable to display.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-ID: <20250311034303.75779-35-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
We have several kinds of sections, and to tell them apart, we use
Section attribute @tag and also the section object's Python type:
type @tag
untagged Section None
@foo: ArgSection 'foo'
Returns: Section 'Returns'
Errors: Section 'Errors'
Since: Section 'Since'
TODO: Section 'TODO'
Note:
* @foo can be a member or a feature description, depending on context.
* tag == 'Since' can be a Since: section or a member or feature
description. If it's a Section, it's the former, and if it's an
ArgSection, it's the latter.
Clean this up as follows. Move the member or feature name to new
ArgSection attribute @name, and replace @tag by enum @kind like this:
type kind name
untagged Section PLAIN
@foo: ArgSection MEMBER 'foo' if member or argument
ArgSection FEATURE 'foo' if feature
Returns: Section RETURNS
Errors: Section ERRORS
Since: Section SINCE
TODO: Section TODO
The qapi-schema tests are updated to account for the new section names;
"TODO" becomes "Todo" and `None` becomes "Plain" there.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-ID: <20250311034303.75779-34-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Instead of using the info object for the doc block as a whole (which
always points to the very first line of the block), update the info
pointer for each call to ensure_untagged_section when the existing
section is otherwise empty. This way, Sphinx error information will
match precisely to where the text actually starts.
For example, this patch will move the info pointer for the "Hello!"
untagged section ...
> ## <-- from here ...
> # Hello! <-- ... to here.
> ##
This doesn't seem to improve error reporting now. It will with the
forthcoming QAPI doc transmogrifier.
If I stick bad rST into qapi/block-core.json like this:
> ##
> # @SnapshotInfo:
> #
> +# rST syntax error: *ahh!
> +#
> # @id: unique shapshot id
> #
> # @name: user chosen name
The existing code's error message will point to the beginning of the doc
comment, which is less than helpful. The transmogrifier's message will
point to the erroneous line, but to accomplish this, it needs this
patch.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-ID: <20250311034303.75779-33-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
We use OrderedDict to ensure dictionary order is insertion order.
Plain dict does that since Python 3.6, but it wasn't guaranteed until
3.7. Since we have 3.7 now, replace OrderedDict by dict.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20250227080757.3978333-3-armbru@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Fully eliminate the "Example" sections in QAPI doc blocks now that they
have all been converted to arbitrary rST syntax using the
".. qmp-example::" directive. Update tests to match.
Migrating to the new syntax
---------------------------
The old "Example:" or "Examples:" section syntax is now caught as an
error, but "Example::" is stil permitted as explicit rST syntax for an
un-lexed, generic preformatted text block.
('Example' is not special in this case, any sentence that ends with "::"
will start an indented code block in rST.)
Arbitrary rST for Examples is now possible, but it's strongly
recommended that documentation authors use the ".. qmp-example::"
directive for consistent visual formatting in rendered HTML docs. The
":title:" directive option may be used to add extra information into the
title bar for the example. The ":annotated:" option can be used to write
arbitrary rST instead, with nested "::" blocks applying QMP formatting
where desired.
Other choices available are ".. code-block:: QMP" which will not create
an "Example:" box, or the short-form "::" code-block syntax which will
not apply QMP highlighting when used outside of the qmp-example
directive.
Why?
----
This patch has several benefits:
1. Example sections can now be written more arbitrarily, mixing
explanatory paragraphs and code blocks however desired.
2. Example sections can now use fully arbitrary rST.
3. All code blocks are now lexed and validated as QMP; increasing
usability of the docs and ensuring validity of example snippets.
(To some extent - This patch only gaurantees it lexes correctly, not
that it's valid under the JSON or QMP grammars. It will catch most
small mistakes, however.)
4. Each qmp-example can be titled or annotated independently without
bypassing the QMP lexer/validator.
(i.e. code blocks are now for *code* only, so we don't have to
sacrifice exposition for having lexically valid examples.)
NOTE: As with the "Notes" conversion (d461c27973), this patch (and the
three preceding) may change the rendering order for Examples in
the current generator. The forthcoming qapidoc rewrite will fix
this by always generating documentation in source order.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-ID: <20240717021312.606116-10-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
The double-colon synax is rST formatting that precedes a literal code
block. We do not want to capture these as QAPI-specific sections.
Coerce blocks that start with e.g. "Example::" to be parsed as untagged
paragraphs instead of special tagged sections.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-ID: <20240626222128.406106-14-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Indentation tweaked for consistency]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
We do not need a dedicated section for notes. By eliminating a specially
parsed section, these notes can be treated as normal rST paragraphs in
the new QMP reference manual, and can be placed and styled much more
flexibly.
Convert all existing "Note" and "Notes" sections to pure rST. As part of
the conversion, capitalize the first letter of each sentence and add
trailing punctuation where appropriate to ensure notes look sensible and
consistent in rendered HTML documentation. Markup is also re-aligned to
the de-facto standard of 3 spaces for directives.
Update docs/devel/qapi-code-gen.rst to reflect the new paradigm, and
update the QAPI parser to prohibit "Note" sections while suggesting a
new syntax. The exact formatting to use is a matter of taste, but a good
candidate is simply:
.. note:: lorem ipsum ...
... dolor sit amet ...
... consectetur adipiscing elit ...
... but there are other choices, too. The Sphinx readthedocs theme
offers theming for the following forms (capitalization unimportant); all
are adorned with a (!) symbol () in the title bar for rendered HTML
docs.
See
https://sphinx-rtd-theme.readthedocs.io/en/stable/demo/demo.html#admonitions
for examples of each directive/admonition in use.
These are rendered in orange:
.. Attention:: ...
.. Caution:: ...
.. WARNING:: ...
These are rendered in red:
.. DANGER:: ...
.. Error:: ...
These are rendered in green:
.. Hint:: ...
.. Important:: ...
.. Tip:: ...
These are rendered in blue:
.. Note:: ...
.. admonition:: custom title
admonition body text
This patch uses ".. note::" almost everywhere, with just two "caution"
directives. Several instances of "Notes:" have been converted to
merely ".. note::", or multiple ".. note::" where appropriate.
".. admonition:: notes" is used in a few places where we had an
ordered list of multiple notes that would not make sense as
standalone/separate admonitions. Two "Note:" following "Example:"
have been turned into ordinary paragraphs within the example.
NOTE: Because qapidoc.py does not attempt to preserve source ordering of
sections, the conversion of Notes from a "tagged section" to an
"untagged section" means that rendering order for some notes *may
change* as a result of this patch. The forthcoming qapidoc.py rewrite
strictly preserves source ordering in the rendered documentation, so
this issue will be rectified in the new generator.
Signed-off-by: John Snow <jsnow@redhat.com>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com> [for block*.json]
Message-ID: <20240626222128.406106-11-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Commit message clarified slightly, period added to one more note]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
If a comment immediately follows a doc block, the parser doesn't ignore
that token appropriately. Fix that.
e.g.
> ##
> # = Hello World!
> ##
>
> # I'm a comment!
will break the parser, because it does not properly ignore the comment
token if it immediately follows a doc block.
Fixes: 3d035cd2cc (qapi: Rewrite doc comment parser)
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20240626222128.406106-7-jsnow@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Change get_doc_indented() to preserve indentation on all subsequent text
lines, and create a compatibility dedent() function for qapidoc.py that
removes indentation the same way get_doc_indented() did.
This is being done for the benefit of a new qapidoc generator which
requires that indentation in argument and features sections are
preserved.
Prior to this patch, a section like this:
```
@name: lorem ipsum
dolor sit amet
consectetur adipiscing elit
```
would have its body text be parsed into:
```
lorem ipsum
dolor sit amet
consectetur adipiscing elit
```
We want to preserve the indentation for even the first body line so that
the entire block can be parsed directly as rST. This patch would now
parse that segment into:
```
lorem ipsum
dolor sit amet
consectetur adipiscing elit
```
This is helpful for formatting arguments and features as field lists in
rST, where the new generator will format this information as:
```
:arg type name: lorem ipsum
dolor sit amet
consectetur apidiscing elit
```
...and can be formed by the simple concatenation of the field list
construct and the body text. The indents help preserve the continuation
of a block-level element, and further allow the use of additional rST
block-level constructs such as code blocks, lists, and other such
markup.
This understandably breaks the existing qapidoc.py; so a new function is
added there to dedent the text for compatibility. Once the new generator
is merged, this function will not be needed any longer and can be
dropped.
I verified this patch changes absolutely nothing by comparing the
md5sums of the QMP ref html pages both before and after the change, so
it's certified inert. QAPI test output has been updated to reflect the
new strategy of preserving indents for rST.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20240626222128.406106-6-jsnow@redhat.com>
[Lost commit message paragraph restored]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20240315152301.3621858-21-armbru@redhat.com>
Dict[str, object] is a stricter type, but with the way that code is
currently arranged, it is infeasible to enforce this strictness.
In particular, although expr.py's entire raison d'être is normalization
and type-checking of QAPI Expressions, that type information is not
"remembered" in any meaningful way by mypy because each individual
expression is not downcast to a specific expression type that holds all
the details of each expression's unique form.
As a result, all of the code in schema.py that deals with actually
creating type-safe specialized structures has no guarantee (myopically)
that the data it is being passed is correct.
There are two ways to solve this:
(1) Re-assert that the incoming data is in the shape we expect it to be, or
(2) Disable type checking for this data.
(1) is appealing to my sense of strictness, but I gotta concede that it
is asinine to re-check the shape of a QAPIExpression in schema.py when
expr.py has just completed that work at length. The duplication of code
and the nightmare thought of needing to update both locations if and
when we change the shape of these structures makes me extremely
reluctant to go down this route.
(2) allows us the chance to miss updating types in the case that types
are updated in expr.py, but it *is* an awful lot simpler and,
importantly, gets us closer to type checking schema.py *at
all*. Something is better than nothing, I'd argue.
So, do the simpler dumber thing and worry about future strictness
improvements later.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20240315152301.3621858-20-armbru@redhat.com>
Shhh!
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20240315152301.3621858-3-armbru@redhat.com>
Small copy-pasto. The correct info field to use in this conditional
block is self.errors.info.
Fixes: 3a025d3d1f
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20240315152301.3621858-2-armbru@redhat.com>
We use section "Returns" for documenting both success and error
response of commands.
I intend to generate better command success response documentation.
Easier when "Returns" documents just he success response.
Create new section tag "Errors". The next two commits will move error
response documentation from "Returns" sections to "Errors" sections.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20240227113921.236097-4-armbru@redhat.com>
Change "'Returns:' is only valid for commands" to "'Returns' section
is only valid for commands".
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20240227113921.236097-3-armbru@redhat.com>
This is chiefly to make code that looks up these sections easier to
read.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20240227113921.236097-2-armbru@redhat.com>
QAPIDoc stores a reference to QAPIParser just to pass it to
QAPIParseError. The resulting error position depends on the state of
the parser. It happens to be the current comment line. Servicable,
but action at a distance.
The commit before previous moved most uses of QAPIParseError from
QAPIDoc to QAPIParser. There are just three left. Convert them to
QAPISemError. This involves passing info to a few methods. Then drop
the reference to QAPIParser.
The three errors lose the column number. Not really interesting here:
it's the comment line's indentation.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20240216145841.2099240-17-armbru@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
The parser recognizes only the first "Features:" line. Any subsequent
ones are treated as ordinary text, as visible in test case
doc-duplicate-features. Recognize "Features:" lines anywhere. A
second one is an error.
A 'Features:' line without any features is useless, but not an error.
Make it an error. This makes detecting a second "Features:" line
easier.
qapi/run-state.json actually has an instance of this since commit
fe17522d85 (qapi: Remove deprecated 'singlestep' member of
StatusInfo). Clean it up.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20240216145841.2099240-16-armbru@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
QAPISchemaParser is a conventional recursive descent parser. Except
QAPISchemaParser.get_doc() delegates most of the doc comment parsing
work to a state machine in QAPIDoc. The state machine doesn't get
tokens like a recursive descent parser, it is fed tokens.
I find this state machine rather opaque and hard to maintain.
Replace it by a conventional parser, all in QAPISchemaParser. Less
code, and (at least in my opinion) easier to understand.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20240216145841.2099240-15-armbru@redhat.com>
Tested-by: Daniel P. Berrangé <berrange@redhat.com>
The parser mostly doesn't create adjacent untagged sections, and
merging the ones it does create is hardly worth the bother. I'm doing
it to avoid behavioral change in the next commit.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20240216145841.2099240-14-armbru@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Putting a blank line before section tags and 'Features:' is good,
existing practice. Enforce it.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20240216145841.2099240-12-armbru@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
By convention, we indent the second and subsequent lines of
descriptions and tagged sections, except for examples.
Turn this into a hard rule, and apply it to examples, too.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20240216145841.2099240-11-armbru@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
[Straightforward conflicts in qapi/migration.json resolved]
docs/devel/qapi-code-gen.txt claims "A heading line must be the first
line of the documentation comment block" since commit
55ec69f8b1 (docs/devel/qapi-code-gen.txt: Update to new rST backend
conventions). Not true, we have code to make it work anywhere in a
free-form doc comment: commit dcdc07a97c (qapi: Make section headings
start a new doc comment block).
Make it true, for simplicity's sake.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20240216145841.2099240-10-armbru@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Since the previous commit, QAPIDoc.Section.name is either
None (untagged section) or the section's tag string ('Returns',
'@name', ...). Rename it to .tag.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20240216145841.2099240-9-armbru@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Improve the message for an empty tagged section from
empty doc section 'Note'
to
text required after 'Note:'
and the message for an empty argument or feature description from
empty doc section 'foo'
to
text required after '@foo:'
Improve the error position to refer to the beginning of the empty
section instead of its end.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20240216145841.2099240-8-armbru@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
When something other than a command has a "Returns" section, the error
message points to the beginning of the definition comment. Point to
the "Returns" section instead.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20240216145841.2099240-7-armbru@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
When documented arguments don't exist, the error message points to the
beginning of the definition comment. Point to the first bogus
argument description instead.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20240216145841.2099240-6-armbru@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
The QAPI generator forces you to document your stuff. Except for
command arguments, event data, and members of enum and object types:
these the generator silently "documents" as "Not documented".
We can't require proper documentation there without first fixing all
the offenders. We've always had too many offenders to pull that off.
Right now, we have more than 500. Worse, we seem to fix old ones no
faster than we add new ones: in the past year, we fixed 22 ones, but
added 26 new ones.
To help arrest the backsliding, make missing documentation an error
unless the command, type, or event is in listed in new pragma
documentation-exceptions.
List all the current offenders: 117 commands and types in qapi/, and 9
in qga/.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20240205074709.3613229-7-armbru@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Conversion of docs/devel/qapi-code-gen.txt to ReST left several
dangling references behind. Fix them to point to
docs/devel/qapi-code-gen.rst.
Fixes: f7aa076dbd (docs: convert qapi-code-gen.txt to ReST)
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20240120095327.666239-4-armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Some very minor housekeeping to make the linters happy once more.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-ID: <20231004230532.3002201-4-jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
The error message is bad when the section is untagged. For instance,
test case doc-interleaved-section produces "'@foobar:' can't follow
'Note' section", which is okay, but if we drop the "Note:" tag, we get
"'@foobar:' can't follow 'None' section, which is bad.
Change the error message to "description of '@foobar:' follows a
section".
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20230510141637.3685080-1-armbru@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
[Conflict with commit 3e32dca3f0 resolved]
Two type hints fail centos-stream-8-x86_64 CI. They are actually
broken. Changing them to Optional[re.Match[str]] fixes them locally
for me, but then CI fails differently. Drop them for now.
Fixes: 3e32dca3f0 (qapi: Rewrite parsing of doc comment section symbols and tags)
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20230517061600.1782455-1-armbru@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Tested-by: Igor Mammedov <imammedo@redhat.com>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20230428105429.1687850-15-armbru@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
The QAPI schema doc comment language provides special syntax for
command and event arguments, struct and union members, alternate
branches, enumeration values, and features: descriptions starting with
"@name:".
By convention, we format them like this:
# @name: Lorem ipsum dolor sit amet, consectetur adipiscing elit,
# sed do eiusmod tempor incididunt ut labore et dolore
# magna aliqua.
Okay for names as short as "name", but we have much longer ones. Their
description gets squeezed against the right margin, like this:
# @dirty-sync-missed-zero-copy: Number of times dirty RAM synchronization could
# not avoid copying dirty pages. This is between
# 0 and @dirty-sync-count * @multifd-channels.
# (since 7.1)
The description text is effectively just 50 characters wide. Easy
enough to read, but can be cumbersome to write.
The awkward squeeze against the right margin makes people go beyond it,
which produces two undesirables: arguments about style, and descriptions
that are unnecessarily hard to read, like this one:
# @postcopy-vcpu-blocktime: list of the postcopy blocktime per vCPU. This is
# only present when the postcopy-blocktime migration capability
# is enabled. (Since 3.0)
We could instead format it like
# @postcopy-vcpu-blocktime:
# list of the postcopy blocktime per vCPU. This is only present
# when the postcopy-blocktime migration capability is
# enabled. (Since 3.0)
or, since the commit before previous, like
# @postcopy-vcpu-blocktime:
# list of the postcopy blocktime per vCPU. This is only present
# when the postcopy-blocktime migration capability is
# enabled. (Since 3.0)
However, I'd rather have
# @postcopy-vcpu-blocktime: list of the postcopy blocktime per vCPU.
# This is only present when the postcopy-blocktime migration
# capability is enabled. (Since 3.0)
because this is how rST field and option lists work.
To get this, we need to let the first non-blank line after the
"@name:" line determine expected indentation.
This fills up the indentation pitfall mentioned in
docs/devel/qapi-code-gen.rst. A related pitfall still exists. Update
the text to show it.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20230428105429.1687850-14-armbru@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
[Work around lack of walrus operator in Python 3.7 and older]
To recognize a line starting with a section symbol and or tag, we
first split it at the first space, then examine the part left of the
space. We can just as well examine the unsplit line, so do that.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20230428105429.1687850-13-armbru@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
[Work around lack of walrus operator in Python 3.7 and older]
When an argument's description starts on the line after the "#arg: "
line, indentation is stripped only from the description's first line,
as demonstrated by the previous commit. Moreover, subsequent lines
with less indentation are not rejected.
Make the first line's indentation the expected indentation for the
remainder of the description. This fixes indentation stripping, and
also requires at least that much indentation.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20230428105429.1687850-12-armbru@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
When the lexer chokes on a stray character, its shows the characters
until the next structural character in the error message. It uses a
regular expression to match a non-empty string of non-structural
characters. Bug: the regular expression treats '"' as structural.
When the lexer chokes on '"', the match fails, and trips
must_match()'s assertion. Fix the regular expression.
Fixes: 14c3279502 (qapi: Improve reporting of lexical errors)
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20230428105429.1687850-4-armbru@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
With the two major JSON-ish type hierarchies clarified for distinct
purposes; QAPIExpression for parsed expressions and JSONValue for
introspection data, remove this FIXME as no longer an action item.
A third JSON-y data type, _ExprValue, is not meant to represent JSON in
the abstract but rather only the possible legal return values from a
single function, get_expr(). It isn't appropriate to attempt to merge it
with either of the above two types.
In theory, it may be possible to define a completely agnostic
one-size-fits-all JSON type hierarchy that any other user could borrow -
in practice, it's tough to wrangle the differences between invariant,
covariant and contravariant types: input and output parameters demand
different properties of such a structure.
However, QAPIExpression serves to authoritatively type user input to the
QAPI parser, while JSONValue serves to authoritatively type qapi
generator *output* to be served back to client users at runtime via
QMP. The AST for these two types are different and cannot be wholly
merged into a unified syntax.
They could, in theory, share some JSON primitive definitions. In
practice, this is currently more trouble than it's worth with mypy's
current expressive power. As such, declare this "done enough for now".
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230215000011.1725012-7-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
We can remove this alias as it only has two usages now, and no longer
pays for the confusion of "yet another type".
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230215000011.1725012-6-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
This patch creates a new type, QAPIExpression, which represents a parsed
expression complete with QAPIDoc and QAPISourceInfo.
This patch turns parser.exprs into a list of QAPIExpression instead,
and adjusts expr.py to match.
This allows the types we specify in parser.py to be "remembered" all the
way through expr.py and into schema.py. Several assertions around
packing and unpacking this data can be removed as a result.
It also corrects a harmless typing error. Before the patch,
check_exprs() allegedly takes a List[_JSONObject]. It actually takes
a list of dicts of the form
{'expr': E, 'info': I, 'doc': D}
where E is of type _ExprValue, I is of type QAPISourceInfo, and D is
of type QAPIDoc. Key 'doc' is optional. This is not a _JSONObject!
Passes type checking anyway, because _JSONObject is Dict[str, object].
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230215000011.1725012-5-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Commit message amended to point out the typing fix]
Eh. Not worth the fuss today. There are bigger fish to fry.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20210930205716.1148693-13-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
The fix for this comment is forthcoming in a future commit, but this
will keep me honest. The linting configuration in ./python/setup.cfg
prohibits 'FIXME' comments. A goal of this long-running series is to
move ./scripts/qapi to ./python/qemu/qapi so that the QAPI generator is
regularly type-checked by GitLab CI.
This comment is a time-bomb to force me to address this issue prior to
that step.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20210930205716.1148693-11-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Annotations do not change runtime behavior.
This commit consists of only annotations.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20210930205716.1148693-10-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Adding static types causes a cycle in the QAPI generator:
[schema -> expr -> parser -> schema]. It exists because the QAPIDoc
class needs the names of types defined by the schema module, but the
schema module needs to import both expr.py/parser.py to do its actual
parsing.
Ultimately, the layering violation is that parser.py should not have any
knowledge of specifics of the Schema. QAPIDoc performs double-duty here
both as a parser *and* as a finalized object that is part of the schema.
In this patch, add the offending type hints alongside the workaround to
avoid the cycle becoming a problem at runtime. See
https://mypy.readthedocs.io/en/latest/runtime_troubles.html#import-cycles
for more information on this workaround technique.
I see three ultimate resolutions here:
(1) Just keep this patch and use the TYPE_CHECKING trick to eliminate
the cycle which is only present during static analysis.
(2) Don't bother to annotate connect_member() et al, give them 'object'
or 'Any'. I don't particularly like this, because it diminishes the
usefulness of type hints for documentation purposes. Still, it's an
extremely quick fix.
(3) Reimplement doc <--> definition correlation directly in schema.py,
integrating doc fields directly into QAPISchemaMember and relieving
the QAPIDoc class of the responsibility. Users of the information
would instead visit the members first and retrieve their
documentation instead of the inverse operation -- visiting the
documentation and retrieving their members.
My preference is (3), but in the short-term (1) is the easiest way to
have my cake (strong type hints) and eat it too (Not have import
cycles). Do (1) for now, but plan for (3).
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20210930205716.1148693-9-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Here's the weird bit. QAPIDoc generally expects -- virtually everywhere
-- that it will always have a current section. The sole exception to
this is in the case that end_comment() is called, which leaves us with
*no* section. However, in this case, we also don't expect to actually
ever mutate the comment contents ever again.
NullSection is just a Null-object that allows us to maintain the
invariant that we *always* have a current section, enforced by static
typing -- allowing us to type that field as QAPIDoc.Section instead of
the more ambiguous Optional[QAPIDoc.Section].
end_section is renamed to switch_section and now accepts as an argument
the new section to activate, clarifying that no callers ever just
unilaterally end a section; they only do so when starting a new section.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20210930205716.1148693-8-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
The "if self._section" clause in end_section is mysterious: In which
circumstances might we end a section when we don't have one?
QAPIDoc always expects there to be a "current section", only except
after a call to end_comment(). This actually *shouldn't* ever be 'None',
so let's remove that logic so I don't wonder why it's like this again in
three months.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20210930205716.1148693-7-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
True, we do not check the validity of this symbol -- but we don't check
the validity of definition names during parse, either -- that happens
later, during the expr check. I don't want to introduce a dependency on
expr.py:check_name_str here and introduce a cycle.
Instead, rest assured that a documentation block is required for each
definition. This requirement uses the names of each section to ensure
that we fulfilled this requirement.
e.g., let's say that block-core.json has a comment block for
"Snapshot!Info" by accident. We'll see this error message:
In file included from ../../qapi/block.json:8:
../../qapi/block-core.json: In struct 'SnapshotInfo':
../../qapi/block-core.json:38: documentation comment is for 'Snapshot!Info'
That's a pretty decent error message.
Now, let's say that we actually mangle it twice, identically:
../../qapi/block-core.json: In struct 'Snapshot!Info':
../../qapi/block-core.json:38: struct has an invalid name
That's also pretty decent. If we forget to fix it in both places, we'll
just be back to the first error.
Therefore, let's just drop this FIXME and adjust the error message to
not imply a more thorough check than is actually performed.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20210930205716.1148693-6-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
John Snow
Markus Armbruster