riscv/qemu - qemu - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Markus Armbruster	df649835fe	qjson: Limit number of tokens in addition to total size Commit `29c75dd` "json-streamer: limit the maximum recursion depth and maximum token count" attempts to guard against excessive heap usage by limiting total token size (it says "token count", but that's a lie). Total token size is a rather imprecise predictor of heap usage: many small tokens use more space than few large tokens with the same input size, because there's a constant per-token overhead: 37 bytes on my system. Tighten this up: limit the token count to 2Mi. Chosen to roughly match the 64MiB total token size limit. Signed-off-by: Markus Armbruster <armbru@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Message-Id: <1448486613-17634-13-git-send-email-armbru@redhat.com>	11 years ago
Paolo Bonzini	9bada89711	qjson: surprise, allocating 6 QObjects per token is expensive Replace the contents of the tokens GQueue with a simple struct. This cuts the amount of memory allocated by tests/check-qjson from ~500MB to ~20MB, and the execution time from 600ms to 80ms on my laptop. Still a lot (some could be saved by using an intrusive list, such as QSIMPLEQ, instead of the GQueue), but the savings are already massive and the right thing to do would probably be to get rid of json-streamer completely. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Message-Id: <1448300659-23559-5-git-send-email-pbonzini@redhat.com> [Straightforwardly rebased on my patches] Signed-off-by: Markus Armbruster <armbru@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com>	11 years ago
Paolo Bonzini	95385fe9ac	qjson: store tokens in a GQueue Even though we still have the "streamer" concept, the tokens can now be deleted as they are read. While doing so convert from QList to GQueue, since the next step will make tokens not a QObject and we will have to do the conversion anyway. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Message-Id: <1448300659-23559-4-git-send-email-pbonzini@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com>	11 years ago
Paolo Bonzini	d2ca7c0b0d	qjson: replace QString in JSONLexer with GString JSONLexer only needs a simple resizable buffer. json-streamer.c can allocate memory for each token instead of relying on reference counting of QStrings. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Message-Id: <1448300659-23559-2-git-send-email-pbonzini@redhat.com> [Straightforwardly rebased on my patches, checkpatch made happy] Signed-off-by: Markus Armbruster <armbru@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com>	11 years ago
Markus Armbruster	c54616608a	qjson: Give each of the six structural chars its own token type Simplifies things, because we always check for a specific one. Signed-off-by: Markus Armbruster <armbru@redhat.com> Message-Id: <1448486613-17634-6-git-send-email-armbru@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com>	11 years ago
Markus Armbruster	0753113a26	qjson: Don't crash when input exceeds nesting limit We limit nesting depth and input size to defend against input triggering excessive heap or stack memory use (commit `29c75dd` json-streamer: limit the maximum recursion depth and maximum token count). However, when the nesting limit is exceeded, parser_context_peek_token()'s assertion fails. Broken in commit `65c0f1e` "json-parser: don't replicate tokens at each level of recursion". To reproduce stuff 1025 open braces or brackets into QMP. Fix by taking the error exit instead of the normal one. Reported-by: Eric Blake <eblake@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Message-Id: <1448486613-17634-3-git-send-email-armbru@redhat.com>	11 years ago
Markus Armbruster	4f2d31fbc0	qjson: Apply nesting limit more sanely The nesting limit from commit `29c75dd` "json-streamer: limit the maximum recursion depth and maximum token count" applies separately to braces and brackets. This makes no sense. Apply it to their sum, because that's actually a measure of recursion depth. Signed-off-by: Markus Armbruster <armbru@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Message-Id: <1448486613-17634-2-git-send-email-armbru@redhat.com>	11 years ago
Paolo Bonzini	a372823a14	build: move qobject files to qobject/ and libqemuutil.a Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>	14 years ago
Paolo Bonzini	7b1b5d1913	qapi: move include files to include/qobject/ Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>	14 years ago
Michael Roth	5e2dafeb19	json-streamer: add handling for JSON_ERROR token/state This allows a JSON_ERROR state to be passed to the streamer to force a flush of the current tokens and pass a NULL token list to the parser rather that have it churn on bad data. (Alternatively we could just not pass it to the parser at all, but it may be useful to push there errors up the stack. NULL token lists are not currently handled by the parser, the next patch will address that) Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com> Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>	15 years ago
Anthony Liguori	eca7db46ff	json-streamer: make sure to reset token_size after emitting a token list Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com> Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>	15 years ago
Anthony Liguori	29c75ddd87	json-streamer: limit the maximum recursion depth and maximum token count Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com> Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>	15 years ago
Anthony Liguori	55f8301f76	json-streamer: allow recovery after bad input Once we detect a malformed message, make sure to reset our state. Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com> Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>	15 years ago
Luiz Capitulino	2e89c06889	json-streamer: Don't use qdict_put_obj() It's not needed, use qobject_put() instead and get a cleaner code. Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>	16 years ago
Anthony Liguori	d7ff3acbb4	Add a JSON message boundary identifier The second stage of our JSON parser is a simple state machine that identifies individual JSON values by counting the levels of nesting of tokens. It does not perform grammar validation. We use this to emit a full JSON value to the parser. Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>	17 years ago

8 Commits (bc92e4e97ed62efa61ba983f57dfe33f10fe1d88)