binutils-gdb

Commit Graph

Author	SHA1	Message	Date
Nick Alcock	483546ce4f	libctf: make ctf_serialize() actually serialize ctf_serialize() evolved from the old ctf_update(), which mutated the in-memory CTF dict to make all the dynamic in-memory types into static, unchanging written-to-the-dict types (by deserializing and reserializing it): back in the days when you could only do type lookups on static types, this meant you could see all the types you added recently, at the small, small cost of making it impossible to change those older types ever again and inducing an amortized O(n^2) cost if you actually wanted to add references to types you added at arbitrary times to later types. It also reset things so that ctf_discard() would throw away only types you added after the most recent ctf_update() call. Some time ago this was all changed so that you could look up dynamic types just as easily as static types: ctf_update() changed so that only its visible side-effect of affecting ctf_discard() remained: the old ctf_update() was renamed to ctf_serialize(), made internal to libctf, and called from the various functions that wrote files out. ... but it was still working by serializing and deserializing the entire dict, swapping out its guts with the newly-serialized copy in an invasive and horrible fashion that coupled ctf_serialize() to almost every field in the ctf_dict_t. This is totally useless, and fixing it is easy: just rip all that code out and have ctf_serialize return a serialized representation, and let everything use that directly. This simplifies most of its callers significantly. (It also points up another bug: ctf_gzwrite() failed to call ctf_serialize() at all, so it would only ever work for a dict you just ctf_write_mem()ed yourself, just for its invisible side-effect of serializing the dict!) This lets us simplify away a bunch of internal-only open-side functionality for overriding the syn_ext_strtab and some just-added functionality for forcing in an existing atoms table, without loss of functionality, and lets us lift the restriction on reserializing a dict that was ctf_open()ed rather than being ctf_create()d: it's now perfectly OK to open a dict, modify it (except for adding members to existing structs, unions, or enums, which fails with -ECTF_RDONLY), and write it out again, just as one would expect. libctf/ * ctf-serialize.c (ctf_symtypetab_sect_sizes): Fix typos. (ctf_type_sect_size): Add static type sizes too. (ctf_serialize): Return the new dict rather than updating the existing dict. No longer fail for dicts with static types; copy them onto the start of the new types table. (ctf_gzwrite): Actually serialize before gzwriting. (ctf_write_mem): Improve forced (test-mode) endian-flipping: flip dicts even if they are too small to be compressed. Improve confusing variable naming. * ctf-archive.c (arc_write_one_ctf): Don't bother to call ctf_serialize: both the functions we call do so. * ctf-string.c (ctf_str_create_atoms): Drop serializing case (atoms arg). * ctf-open.c (ctf_simple_open): Call ctf_bufopen directly. (ctf_simple_open_internal): Delete. (ctf_bufopen_internal): Delete/rename to ctf_bufopen: no longer bother with syn_ext_strtab or forced atoms table, serialization no longer needs them. * ctf-create.c (ctf_create): Call ctf_bufopen directly. * ctf-impl.h (ctf_str_create_atoms): Drop atoms arg. (ctf_simple_open_internal): Delete. (ctf_bufopen_internal): Likewise. (ctf_serialize): Adjust. * testsuite/libctf-lookup/add-to-opened.c: Adjust now that this is supposed to work.	2 years ago
Nick Alcock	cf9da3b0b6	libctf: rethink strtab writeout This commit finally adjusts strtab writeout so that repeated writeouts, or writeouts of a dict that was read in earlier, only sorts the portion of the strtab that was newly added. There are three intertwined changes here: - pull the contents of strtabs from newly ctf_bufopened dicts into the atoms table, so that future additions will reuse the existing offset etc rather than adding new identical strings - allow the internal ctf_bufopen done by serialization to contribute its existing atoms table, so that existing atoms can be used for the remainder of the open process (like name table construction): this atoms table currente gets thrown away in the mass reassignment done later in ctf_serialize in any case, but it needs to be there during the open. - rewrite ctf_str_write_strtab so that a) it uses iterators rather than ctf__iter, reducing pointless structures which serve no other purpose than to implement ordinary variable scope, but more clunkily, and b) retains the existing strtab on the front of the new one, with its sort retained, rather than resorting, so all existing already-written strtab offsets remain valid across the call. This latter change finally permits repeated serializations, and reserializations of ctf_open()ed dicts, to work, but for now we keep the code that prevents that because serialization is about to change again in a way that will make it more obvious that doing such things is safe, and we can take it out then. (There are also some smaller changes like moving the purge of the refs table into ctf_str_write_strtab(), since that's where the changes happen that invalidate it, rather than doing it in ctf_serialize(). We also prohibit something that has never worked, opening a dict and then reporting symbols to it via ctf_link_add_strtab() et al: you must do that to newly-created dicts which have had stuff ctf_link()ed into them. This is very unlikely ever to be a problem in practice: linkers just don't do that sort of thing.) libctf/ ctf-create.c (ctf_create): Add (temporary) atoms arg. * ctf-impl.h (struct ctf_dict.ctf_dynstrtab): New. (ctf_str_create_atoms): Adjust. (ctf_str_write_strtab): Likewise. (ctf_simple_open_internal): Likewise. * ctf-open.c (ctf_simple_open_internal): Add atoms arg. (ctf_bufopen): Likewise. (ctf_bufopen_internal): Initialize just enough of an atoms table: pre-init from the atoms arg if supplied. (ctf_simple_open): Adjust. * ctf-serialize.c (ctf_serialize): Constify the strtab. Move ref list purging into ctf_str_write_strtab. Initialize the new dict with the old dict's atoms table. Accept the new strtab from ctf_str_write_strtab. Adjust for addition of ctf_dynstrtab. * ctf-string.c (ctf_strraw_explicit): Improve comments. (ctf_str_create_atoms): Prepopulate from an existing atoms table, or alternatively pull in all strings from the strtab and turn them into atoms. (ctf_str_free_atoms): Free the dynstrtab and its strtab. (struct ctf_strtab_write_state): Remove. (ctf_str_count_strtab): Fold this... (ctf_str_populate_sorttab): ... and this... (ctf_str_write_strtab): ... into this. Prepend existing strings to the strtab rather than resorting them (and wrecking their offsets). Keep the dynstrtab updated. Update refs for all atoms with refs, whether or not they are strings newly added to the strtab.	2 years ago
Nick Alcock	629acbe4a3	libctf: rename ctf_dict.ctf_{symtab,strtab} These two fields are constantly confusing because CTF dicts contain both a symtypetab and strtab, but these fields are not that: they are the symtab and strtab from the ELF file. We have enough string tables now (internal, external, synthetic external, dynamic) that we need to at least name them better than this to avoid getting totally confused. Rename them to ctf_ext_symtab and ctf_ext_strtab. libctf/ * ctf-dump.c (ctf_dump_objts): Rename ctf_symtab -> ctf_ext_symtab. * ctf-impl.h (struct ctf_dict.ctf_symtab): Rename to... (struct ctf_dict.ctf_ext_strtab): ... this. (struct ctf_dict.ctf_strtab): Rename to... (struct ctf_dict.ctf_ext_strtab): ... this. * ctf-lookup.c (ctf_lookup_symbol_name): Adapt. (ctf_lookup_symbol_idx): Adapt. (ctf_lookup_by_sym_or_name): Adapt. * ctf-open.c (ctf_bufopen_internal): Adapt. (ctf_dict_close): Adapt. (ctf_getsymsect): Adapt. (ctf_getstrsect): Adapt. (ctf_symsect_endianness): Adapt.	3 years ago
Nick Alcock	8a60c93096	libctf: support addition of types to dicts read via ctf_open() libctf has long declared deserialized dictionaries (out of files or ELF sections or memory buffers or whatever) to be read-only: back in the furthest prehistory this was not the case, in that you could add a few sorts of type to such dicts, but attempting to do so often caused horrible memory corruption, so I banned the lot. But it turns out real consumers want it (notably DTrace, which synthesises pointers to types that don't have them and adds them to the ctf_open()ed dicts if it needs them). Let's bring it back again, but without the memory corruption and without the massive code duplication required in days of yore to distinguish between static and dynamic types: the representation of both types has been identical for a few years, with the only difference being that types as a whole are stored in a big buffer for types read in via ctf_open and per-type hashtables for newly-added types. So we discard the internally-visible concept of "readonly dictionaries" in favour of declaring the range of types that were already present when the dict was read in to be read-only: you can't modify them (say, by adding members to them if they're structs, or calling ctf_set_array on them), but you can add more types and point to them. (The API remains the same, with calls sometimes returning ECTF_RDONLY, but now they do so less often.) This is a fairly invasive change, mostly because code written since the ban was introduced didn't take the possibility of a static/dynamic split into account. Some of these irregularities were hard to define as anything but bugs. Notably: - The symbol handling was assuming that symbols only needed to be looked for in dynamic hashtabs or static linker-laid-out indexed/ nonindexed layouts, but now we want to check both in case people added more symbols to a dict they opened. - The code that handles type additions wasn't checking to see if types with the same name existed at all (so you could do ctf_add_typedef (fp, "foo", bar) repeatedly without error). This seems reasonable for types you just added, but we probably do want to ban addition of types with names that override names we already used in the ctf_open()ed portion, since that would probably corrupt existing type relationships. (Doing things this way also avoids causing new errors for any existing code that was doing this sort of thing.) - ctf_lookup_variable entirely failed to work for variables just added by ctf_add_variable: you had to write the dict out and read it back in again before they appeared. - The symbol handling remembered what symbols you looked up but didn't remember their types, so you could look up an object symbol and then find it popping up when you asked for function symbols, which seems less than ideal. Since we had to rejig things enough to be able to distinguish function and object symbols internally anyway (in order to give suitable errors if you try to add a symbol with a name that already existed in the ctf_open()ed dict), this bug suddenly became more visible and was easily fixed. We do not (yet) support writing out dicts that have been previously read in via ctf_open() or other deserializer (you can look things up in them, but not write them out a second time). This never worked, so there is no incompatibility; if it is needed at a later date, the serializer is a little bit closer to having it work now (the only table we don't deal with is the types table, and that's because the upcoming CTFv4 changes are likely to make major changes to the way that table is represented internally, so adding more code that depends on its current form seems like a bad idea). There is a new testcase that tests much of this, in particular that modification of existing types is still banned and that you can add new ones and chase them without error. libctf/ * ctf-impl.h (struct ctf_dict.ctf_symhash): Split into... (ctf_dict.ctf_symhash_func): ... this and... (ctf_dict.ctf_symhash_objt): ... this. (ctf_dict.ctf_stypes): New, counts static types. (LCTF_INDEX_TO_TYPEPTR): Use it instead of CTF_RDWR. (LCTF_RDWR): Deleted. (LCTF_DIRTY): Renumbered. (LCTF_LINKING): Likewise. (ctf_lookup_variable_here): New. (ctf_lookup_by_sym_or_name): Likewise. (ctf_symbol_next_static): Likewise. (ctf_add_variable_forced): Likewise. (ctf_add_funcobjt_sym_forced): Likewise. (ctf_simple_open_internal): Adjust. (ctf_bufopen_internal): Likewise. * ctf-create.c (ctf_grow_ptrtab): Adjust a lot to start with. (ctf_create): Migrate a bunch of initializations into bufopen. Force recreation of name tables. Do not forcibly override the model, let ctf_bufopen do it. (ctf_static_type): New. (ctf_update): Drop LCTF_RDWR check. (ctf_dynamic_type): Likewise. (ctf_add_function): Likewise. (ctf_add_type_internal): Likewise. (ctf_rollback): Check ctf_stypes, not LCTF_RDWR. (ctf_set_array): Likewise. (ctf_add_struct_sized): Likewise. (ctf_add_union_sized): Likewise. (ctf_add_enum): Likewise. (ctf_add_enumerator): Likewise (only on the target dict). (ctf_add_member_offset): Likewise. (ctf_add_generic): Drop LCTF_RDWR check. Ban addition of types with colliding names. (ctf_add_forward): Note safety under the new rules. (ctf_add_variable): Split all but the existence check into... (ctf_add_variable_forced): ... this new function. (ctf_add_funcobjt_sym): Likewise... (ctf_add_funcobjt_sym_forced): ... for this new function. * ctf-link.c (ctf_link_add_linker_symbol): Ban calling on dicts with any stypes. (ctf_link_add_strtab): Likewise. (ctf_link_shuffle_syms): Likewise. (ctf_link_intern_extern_string): Note pre-existing prohibition. * ctf-lookup.c (ctf_lookup_by_id): Drop LCTF_RDWR check. (ctf_lookup_variable): Split out looking in a dict but not its parent into... (ctf_lookup_variable_here): ... this new function. (ctf_lookup_symbol_idx): Track whether looking up a function or object: cache them separately. (ctf_symbol_next): Split out looking in non-dynamic symtypetab entries to... (ctf_symbol_next_static): ... this new function. Don't get confused by the simultaneous presence of static and dynamic symtypetab entries. (ctf_try_lookup_indexed): Don't waste time looking up symbols by index before there can be any idea how symbols are numbered. (ctf_lookup_by_sym_or_name): Distinguish between function and data object lookups. Drop LCTF_RDWR. (ctf_lookup_by_symbol): Adjust. (ctf_lookup_by_symbol_name): Likewise. * ctf-open.c (init_types): Rename to... (init_static_types): ... this. Drop LCTF_RDWR. Populate ctf_stypes. (ctf_simple_open): Drop writable arg. (ctf_simple_open_internal): Likewise. (ctf_bufopen): Likewise. (ctf_bufopen_internal): Populate fields only used for writable dicts. Drop LCTF_RDWR. (ctf_dict_close): Cater for symhash cache split. * ctf-serialize.c (ctf_serialize): Use ctf_stypes, not LCTF_RDWR. * ctf-types.c (ctf_variable_next): Drop LCTF_RDWR. * testsuite/libctf-lookup/add-to-opened*: New test.	3 years ago
Nick Alcock	2ba5ec13b2	libctf: fix name lookup in dicts containing base-type bitfields The intent of the name lookup code was for lookups to yield non-bitfield basic types except if none existed with a given name, and only then return bitfield types with that name. Unfortunately, the code as written only does this if the base type has a type ID higher than all bitfield types, which is most unlikely (the opposite is almost always the case). Adjust it so that what ends up in the name table is the highest-width zero-offset type with a given name, if any such exist, and failing that the first type with that name we see, no matter its offset. (We don't define which bitfield type you get, after all, so we might as well just stuff in the first we find.) Reported by Stephen Brennan <stephen.brennan@oracle.com>. libctf/ * ctf-open.c (init_types): Modify to allow some lookups during open; detect bitfield name reuse and prefer less bitfieldy types. * testsuite/libctf-writable/libctf-bitfield-name-lookup.*: New test.	3 years ago
Nick Alcock	54a0219150	libctf: remove static/dynamic name lookup distinction libctf internally maintains a set of hash tables for type name lookups, one for each valid C type namespace (struct, union, enum, and everything else). Or, rather, it maintains two sets of hash tables: one, a ctf_hash , is meant for lookups in ctf_(buf)open()ed dicts with fixed content; the other, a ctf_dynhash , is meant for lookups in ctf_create()d dicts. This distinction was somewhat valuable in the far pre-binutils past when two different hashtable implementations were used (one expanding, the other fixed-size), but those days are long gone: the hash table implementations are almost identical, both wrappers around the libiberty hashtab. The ctf_dynhash has many more capabilities than the ctf_hash (iteration, deletion, etc etc) and has no downsides other than starting at a fixed, arbitrary small size. That limitation is easy to lift (via a new ctf_dynhash_create_sized()), following which we can throw away nearly all the ctf_hash implementation, and all the code to choose between readable and writable hashtabs; the few convenience functions that are still useful (for insertion of name -> type mappings) can also be generalized a bit so that the extra string verification they do is potentially available to other string lookups as well. (libctf still has two hashtable implementations, ctf_dynhash, above, and ctf_dynset, which is a key-only hashtab that can avoid a great many malloc()s, used for high-volume applications in the deduplicator.) libctf/ * ctf-create.c (ctf_create): Eliminate ctn_writable. (ctf_dtd_insert): Likewise. (ctf_dtd_delete): Likewise. (ctf_rollback): Likewise. (ctf_name_table): Eliminate ctf_names_t. * ctf-hash.c (ctf_dynhash_create): Comment update. Reimplement in terms of... (ctf_dynhash_create_sized): ... this new function. (ctf_hash_create): Remove. (ctf_hash_size): Remove. (ctf_hash_define_type): Remove. (ctf_hash_destroy): Remove. (ctf_hash_lookup_type): Rename to... (ctf_dynhash_lookup_type): ... this. (ctf_hash_insert_type): Rename to... (ctf_dynhash_insert_type): ... this, moving validation to... * ctf-string.c (ctf_strptr_validate): ... this new function. * ctf-impl.h (struct ctf_names): Extirpate. (struct ctf_lookup.ctl_hash): Now a ctf_dynhash_t. (struct ctf_dict): All ctf_names_t fields are now ctf_dynhash_t. (ctf_name_table): Now returns a ctf_dynhash_t. (ctf_lookup_by_rawhash): Remove. (ctf_hash_create): Likewise. (ctf_hash_insert_type): Likewise. (ctf_hash_define_type): Likewise. (ctf_hash_lookup_type): Likewise. (ctf_hash_size): Likewise. (ctf_hash_destroy): Likewise. (ctf_dynhash_create_sized): New. (ctf_dynhash_insert_type): New. (ctf_dynhash_lookup_type): New. (ctf_strptr_validate): New. * ctf-lookup.c (ctf_lookup_by_name_internal): Adapt. * ctf-open.c (init_types): Adapt. (ctf_set_ctl_hashes): Adapt. (ctf_dict_close): Adapt. * ctf-serialize.c (ctf_serialize): Adapt. * ctf-types.c (ctf_lookup_by_rawhash): Remove.	3 years ago
Alan Modra	fd67aa1129	Update year range in copyright notice of binutils files Adds two new external authors to etc/update-copyright.py to cover bfd/ax_tls.m4, and adds gprofng to dirs handled automatically, then updates copyright messages as follows: 1) Update cgen/utils.scm emitted copyrights. 2) Run "etc/update-copyright.py --this-year" with an extra external author I haven't committed, 'Kalray SA.', to cover gas testsuite files (which should have their copyright message removed). 3) Build with --enable-maintainer-mode --enable-cgen-maint=yes. 4) Check out /po/.pot which we don't update frequently.	3 years ago
Alan Modra	027333da75	ctf segfaults PR 30228 PR 30229 * ctf-open.c (ctf_bufopen_internal): Check for NULL cts_data. * ctf-archive.c (ctf_arc_bufpreamble, ctf_arc_bufopen): Likewise.	3 years ago
Alan Modra	d87bef3a7b	Update year range in copyright notice of binutils files The newer update-copyright.py fixes file encoding too, removing cr/lf on binutils/bfdtest2.c and ld/testsuite/ld-cygwin/exe-export.exp, and embedded cr in binutils/testsuite/binutils-all/ar.exp string match.	4 years ago
Nick Alcock	faf5e6ace8	libctf: add LIBCTF_WRITE_FOREIGN_ENDIAN debugging option libctf has always handled endianness differences by detecting foreign-endian CTF dicts on the input and endian-flipping them: dicts are always written in native endianness. This makes endian-awareness very low overhead, but it means that the foreign-endian code paths almost never get routinely tested, since "make check" usually reads in dicts ld has just written out: only a few corrupted-CTF tests are actually in fixed endianness, and even they only test the foreign- endian code paths when you run make check on a big-endian machine. (And the fix is surely not to add more .s-based tests like that, because they are a nightmare to maintain compared to the C-code-based ones.) To improve on this, add a new environment variable, LIBCTF_WRITE_FOREIGN_ENDIAN, which causes libctf to unconditionally endian-flip at ctf_write time, so the output is always in the wrong endianness. This then tests the foreign-endian read paths properly at open time. Make this easier by restructuring the writeout code in ctf-serialize.c, which duplicates the maybe-gzip-and-write-out code three times (once for ctf_write_mem, with thresholding, and once each for ctf_compress_write and ctf_write just so those can avoid thresholding and/or compression). Instead, have the latter two call the former with thresholds of 0 or (size_t) -1, respectively. The endian-flipping code itself gains a bit of complexity, because one single endian-flipper (flip_types) was assuming the input to be in foreign-endian form and assuming it could pull things out of the input once they had been flipped and make sense of them. At the cost of a few lines of duplicated initializations, teach it to read before flipping if we're flipping to foreign-endianness instead of away from it. libctf/ * ctf-impl.h (ctf_flip_header): No longer static. (ctf_flip): Likewise. * ctf-open.c (flip_header): Rename to... (ctf_flip_header): ... this, now it is not private to one file. (flip_ctf): Rename... (ctf_flip): ... this too. Add FOREIGN_ENDIAN arg. (flip_types): Likewise. Use it. (ctf_bufopen_internal): Adjust calls. * ctf-serialize.c (ctf_write_mem): Add flip_endian path via a newly-allocated bounce buffer. (ctf_compress_write): Move below ctf_write_mem and reimplement in terms of it. (ctf_write): Likewise. (ctf_gzwrite): Note that this obscure writeout function does not support endian-flipping.	4 years ago
Nick Alcock	84f5c557a4	libctf, ld: diagnose corrupted CTF header cth_strlen The last section in a CTF dict is the string table, at an offset represented by the cth_stroff header field. Its length is recorded in the next field, cth_strlen, and the two added together are taken as the size of the CTF dict. Upon opening a dict, we check that none of the header offsets exceed this size, and we check when uncompressing a compressed dict that the result of the uncompression is the same length: but CTF dicts need not be compressed, and short ones are not. Uncompressed dicts just use the ctf_size without checking it. This field is thankfully almost unused: it is mostly used when reserializing a dict, which can't be done to dicts read off disk since they're read-only. However, when opening an uncompressed foreign-endian dict we have to copy it out of the mmaped region it is stored in so we can endian- swap it, and we use ctf_size when doing that. When the cth_strlen is corrupt, this can overrun. Fix this by checking the ctf_size in all uncompressed cases, just as we already do in the compressed case. Add a new test. This came to light because various corrupted-CTF raw-asm tests had an incorrect cth_strlen: fix all of them so they produce the expected error again. libctf/ PR libctf/28933 * ctf-open.c (ctf_bufopen_internal): Always check uncompressed CTF dict sizes against the section size in case the cth_strlen is corrupt. ld/ PR libctf/28933 * testsuite/ld-ctf/diag-strlen-invalid.: New test, derived from diag-cttname-invalid.s. testsuite/ld-ctf/diag-cttname-invalid.s: Fix incorrect cth_strlen. * testsuite/ld-ctf/diag-cttname-null.s: Likewise. * testsuite/ld-ctf/diag-cuname.s: Likewise. * testsuite/ld-ctf/diag-parlabel.s: Likewise. * testsuite/ld-ctf/diag-parname.s: Likewise.	4 years ago
Alan Modra	a2c5833233	Update year range in copyright notice of binutils files The result of running etc/update-copyright.py --this-year, fixing all the files whose mode is changed by the script, plus a build with --enable-maintainer-mode --enable-cgen-maint=yes, then checking out /po/.pot which we don't update frequently. The copy of cgen was with commit d1dd5fcc38ead reverted as that commit breaks building of bfp opcodes files.	5 years ago
Nick Alcock	b62d5edd0a	libctf: fix handling of CTF symtypetab sections emitted by older GCC Older (pre-upstreaming) GCC emits a function symtypetab section of a format never read by any extant libctf. We can detect such CTF dicts by the lack of the CTF_F_NEWFUNCINFO flag in their header, and we do so when reading in the symtypetab section -- but if the set of symbols with types is sufficiently sparse, even an older GCC will emit a function index section. In NEWFUNCINFO-capable compilers, this section will always be the exact same length as the corresponding function section (each is an array of uint32_t, associated 1:1 with each other). But this is not true for the older compiler, for which the sections are different lengths. We check to see if the function symtypetab section and its index are the same length, but we fail to skip this check when this is not a NEWFUNCINFO dict, and emit a spurious corruption error for a CTF dict we could have perfectly well opened and used. Fix trivial: check the flag (and fix the terrible grammar of the error message at the same time). libctf/ChangeLog 2021-09-27 Nick Alcock <nick.alcock@oracle.com> * ctf-open.c (ctf_bufopen_internal): Don't complain about corrupt function index symtypetab sections if this is an old-format function symtypetab section (which should be ignored in any case). Fix bad grammar.	5 years ago
Alan Modra	174fe10cb6	ubsan: libctf: applying zero offset to null pointer * ctf-open.c (init_symtab): Avoid ubsan error.	5 years ago
Nick Alcock	49da556c65	libctf, include: support an alternative encoding for nonrepresentable types Before now, types that could not be encoded in CTF were represented as references to type ID 0, which does not itself appear in the dictionary. This choice is annoying in several ways, principally that it forces generators and consumers of CTF to grow special cases for types that are referenced in valid dicts but don't appear. Allow an alternative representation (which will become the only representation in format v4) whereby nonrepresentable types are encoded as actual types with kind CTF_K_UNKNOWN (an already-existing kind theoretically but not in practice used for padding, with value 0). This is backward-compatible, because CTF_K_UNKNOWN was not used anywhere before now: it was used in old-format function symtypetabs, but these were never emitted by any compiler and the code to handle them in libctf likely never worked and was removed last year, in favour of new-format symtypetabs that contain only type IDs, not type kinds. In order to link this type, we need an API addition to let us add types of unknown kind to the dict: we let them optionally have names so that GCC can emit many different unknown types and those types with identical names will be deduplicated together. There are also small tweaks to the deduplicator to actually dedup such types, to let opening of dicts with unknown types with names work, to return the ECTF_NONREPRESENTABLE error on resolution of such types (like ID 0), and to print their names as something useful but not a valid C identifier, mostly for the sake of the dumper. Tests added in the next commit. include/ChangeLog 2021-05-06 Nick Alcock <nick.alcock@oracle.com> * ctf.h (CTF_K_UNKNOWN): Document that it can be used for nonrepresentable types, not just padding. * ctf-api.h (ctf_add_unknown): New. libctf/ChangeLog 2021-05-06 Nick Alcock <nick.alcock@oracle.com> * ctf-open.c (init_types): Unknown types may have names. * ctf-types.c (ctf_type_resolve): CTF_K_UNKNOWN is as non-representable as type ID 0. (ctf_type_aname): Print unknown types. * ctf-dedup.c (ctf_dedup_hash_type): Do not early-exit for CTF_K_UNKNOWN types: they have real hash values now. (ctf_dedup_rwalk_one_output_mapping): Treat CTF_K_UNKNOWN types like other types with no referents: call the callback and do not skip them. (ctf_dedup_emit_type): Emit via... * ctf-create.c (ctf_add_unknown): ... this new function. * libctf.ver (LIBCTF_1.2): Add it.	5 years ago
Nick Alcock	e4c78f303d	libctf: a couple of small error-handling fixes Out-of-memory errors initializing the string atoms table were disregarded (though they would have caused a segfault very shortly afterwards). Errors hashing types during deduplication were only reported if they happened on the output dict, which is almost never the case (most errors are going to be on the dict we're working over, which is going to be one of the inputs). (The error was detected in both cases, but the errno was extracted from the wrong dict.) libctf/ChangeLog 2021-03-18 Nick Alcock <nick.alcock@oracle.com> * ctf-dedup.c (ctf_dedup_rhash_type): Report errors on the input dict properly. * ctf-open.c (ctf_bufopen_internal): Report errors initializing the atoms table.	5 years ago
Nick Alcock	f4f60336da	libctf, include: find types of symbols by name The existing ctf_lookup_by_symbol and ctf_arc_lookup_symbol functions suffice to look up the types of symbols if the caller already has a symbol number. But the caller often doesn't have one of those and only knows the name of the symbol: also, in object files, the caller might not have a useful symbol number in any sense (and neither does libctf: the 'symbol number' we use in that case literally starts at 0 for the lexicographically first-sorted symbol in the symtypetab and counts those symbols, so it corresponds to nothing useful). This means that even though object files have a symtypetab (generated by the compiler or by ld -r), the only way we can look up anything in it is to iterate over all symbols in turn with ctf_symbol_next until we find the one we want. This is unhelpful and pointlessly inefficient. So add a pair of functions to look up symbols by name in a dict and in a whole archive: ctf_lookup_by_symbol_name and ctf_arc_lookup_symbol_name. These are identical to the existing functions except that they take symbol names rather than symbol numbers. To avoid insane repetition, we do some refactoring in the process, so that both ctf_lookup_by_symbol and ctf_arc_lookup_symbol turn into thin wrappers around internal functions that do both lookup by symbol index and lookup by name. This massively reduces code duplication because even the existing lookup-by-index stuff wants to use a name sometimes (when looking up in indexed sections), and the new lookup-by-name stuff has to turn it into an index sometimes (when looking up in non-indexed sections): doing it this way lets us share most of that. The actual name->index lookup is done by ctf_lookup_symbol_idx. We do not anticipate this lookup to be as heavily used as ld.so symbol lookup by many orders of magnitude, so using the ELF symbol hashes would probably take more time to read them than is saved by using the hashes, and it adds a lot of complexity. Instead, do a linear search for the symbol name, caching all the name -> index mappings as we go, so that future searches are likely to hit in the cache. To avoid having to repeat this search over and over in a CTF archive when ctf_arc_lookup_symbol_name is used, have cached archive lookups (the sort done by ctf_arc_lookup_symbol* and the ctf_archive_next iterator) pick out the first dict they cache in a given archive and store it in a new ctf_archive field, ctfi_crossdict_cache. This can be used to store cross-dictionary cached state that depends on things like the ELF symbol table rather than the contents of any one dict. ctf_lookup_symbol_idx then caches its name->index mappings in the dictionary named in the crossdict cache, if any, so that ctf_lookup_symbol_idx in other dicts in the same archive benefit from the previous linear search, and the symtab only needs to be scanned at most once. (Note that if you call ctf_lookup_by_symbol_name in one specific dict, and then follow it with a ctf_arc_lookup_symbol_name, the former will not use the crossdict cache because it's only populated by the dict opens in ctf_arc_lookup_symbol_name. This is harmless except for a small one-off waste of memory and time: it's only a cache, after all. We can fix this later by using the archive caching machinery more aggressively.) In ctf-archive, we do similar things, turning ctf_arc_lookup_symbol into a wrapper around a new function that does both index -> ID and name -> ID lookups across all dicts in an archive. We add a new ctfi_symnamedicts cache that maps symbol names to the ctf_dict_t * that it was found in (so that linear searches for symbols don't need to be repeated): but we also remove a cache, the ctfi_syms cache that was memoizing the actual ctf_id_t returned from every call to ctf_arc_lookup_symbol. This is pointless: all it saves is one call to ctf_lookup_by_symbol, and that's basically an array lookup and nothing more so isn't worth caching. (Equally, given that symbol -> index mappings are cached by ctf_lookup_by_symbol_name, those calls are nearly free after the first call, so there's no point caching the ctf_id_t in that case either.) We fix up one test that was doing manual symbol lookup to use ctf_arc_lookup_symbol instead, and enhance it to check that the caching layer is not totally broken: we also add a new test to do lookups in a .o file, and another to do lookups in an archive with conflicted types and make sure that sort of multi-dict lookup is actually working. include/ChangeLog 2021-02-17 Nick Alcock <nick.alcock@oracle.com> * ctf-api.h (ctf_arc_lookup_symbol_name): New. (ctf_lookup_by_symbol_name): Likewise. libctf/ChangeLog 2021-02-17 Nick Alcock <nick.alcock@oracle.com> * ctf-impl.h (ctf_dict_t) <ctf_symhash>: New. <ctf_symhash_latest>: Likewise. (struct ctf_archive_internal) <ctfi_crossdict_cache>: New. <ctfi_symnamedicts>: New. <ctfi_syms>: Remove. (ctf_lookup_symbol_name): Remove. * ctf-lookup.c (ctf_lookup_symbol_name): Propagate errors from parent properly. Make static. (ctf_lookup_symbol_idx): New, linear search for the symbol name, cached in the crossdict cache's ctf_symhash (if available), or this dict's (otherwise). (ctf_try_lookup_indexed): Allow the symname to be passed in. (ctf_lookup_by_symbol): Turn into a wrapper around... (ctf_lookup_by_sym_or_name): ... this, supporting name lookup too, using ctf_lookup_symbol_idx in non-writable dicts. Special-case name lookup in dynamic dicts without reported symbols, which have no symtab or dynsymidx but where name lookup should still work. (ctf_lookup_by_symbol_name): New, another wrapper. * ctf-archive.c (enosym): Note that this is present in ctfi_symnamedicts too. (ctf_arc_close): Adjust for removal of ctfi_syms. Free the ctfi_symnamedicts. (ctf_arc_flush_caches): Likewise. (ctf_dict_open_cached): Memoize the first cached dict in the crossdict cache. (ctf_arc_lookup_symbol): Turn into a wrapper around... (ctf_arc_lookup_sym_or_name): ... this. No longer cache ctf_id_t lookups: just call ctf_lookup_by_symbol as needed (but still cache the dicts those lookups succeed in). Add lookup-by-name support, with dicts of successful lookups cached in ctfi_symnamedicts. Refactor the caching code a bit. (ctf_arc_lookup_symbol_name): New, another wrapper. * ctf-open.c (ctf_dict_close): Free the ctf_symhash. * libctf.ver (LIBCTF_1.2): New version. Add ctf_lookup_by_symbol_name, ctf_arc_lookup_symbol_name. * testsuite/libctf-lookup/enum-symbol.c (main): Use ctf_arc_lookup_symbol rather than looking up the name ourselves. Fish it out repeatedly, to make sure that symbol caching isn't broken. (symidx_64): Remove. (symidx_32): Remove. * testsuite/libctf-lookup/enum-symbol-obj.lk: Test symbol lookup in an unlinked object file (indexed symtypetab sections only). * testsuite/libctf-writable/symtypetab-nonlinker-writeout.c (try_maybe_reporting): Check symbol types via ctf_lookup_by_symbol_name as well as ctf_symbol_next. * testsuite/libctf-lookup/conflicting-type-syms.*: New test of lookups in a multi-dict archive.	6 years ago
Nick Alcock	78f28b89e8	libctf: rip out dead code handling typedefs with no name There is special code in libctf to handle typedefs with no name, which the code calls "anonymous typedef nodes". These monsters are obviously not something C programs can include: the whole point of a ttypedef is to introduce a new name. Looking back at the history of DWARF in GCC, the only thing (outside C++ anonymous namespaces) which can generate a DW_TAG_typedef without a DW_AT_name is obsolete code to handle the long-removed -feliminate-dwarf2-dups option. Looking at OpenSolaris, typedef nodes with no name couldn't be generated by the DWARF->CTF converter at all (and its deduplicator barfed on them): the only reason for the existence of this code is a special case working around a peculiarity of stabs whereby types could sometimes be referenced before they were introduced. We don't need to carry code in libctf to handle special cases in an obsolete OpenSolaris converter (that yields a format that isn't readable by libctf anyway). So drop it. libctf/ChangeLog 2021-01-27 Nick Alcock <nick.alcock@oracle.com> * ctf-open.c (init_types): Rip out code to check anonymous typedef nodes. * ctf-create.c (ctf_add_reftype): Likewise. * ctf-lookup.c (refresh_pptrtab): Likewise.	6 years ago
Nick Alcock	abe4ca69a1	libctf: fix lookups of pointers by name in parent dicts When you look up a type by name using ctf_lookup_by_name, in most cases libctf can just strip off any qualifiers and look for the name, but for pointer types this doesn't work, since the caller will want the pointer type itself. But pointer types are nameless, and while they cite the types they point to, looking up a type by name requires a link going the other way, from the type pointed to to the pointer type that points to it. libctf has always built this up at open time: ctf_ptrtab is an array of type indexes pointing from the index of every type to the index of the type that points to it. But because it is built up at open time (and because it uses type indexes and not type IDs) it is restricted to working within a single dict and ignoring parent/child relationships. This is normally invisible, unless you manage to get a dict with a type in the parent but the only pointer to it in a child. The ctf_ptrtab will not track this relationship, so lookups of this pointer type by name will fail. Since which type is in the parent and which in the child is largely opaque to the user (which goes where is up to the deduplicator, and it can and does reshuffle things to save space), this leads to a very bad user experience, with an obviously-visible pointer type which ctf_lookup_by_name claims doesn't exist. The fix is to have another array, ctf_pptrtab, which is populated in child dicts: like the parent's ctf_ptrtab, it has one element per type in the parent, but is all zeroes except for those types which are pointed to by types in the child: so it maps parent dict indices to child dict indices. The array is grown, and new child types scanned, whenever a lookup happens and new types have been added to the child since the last time a lookup happened that might need the pptrtab. (So for non-writable dicts, this only happens once, since new types cannot be added to non-writable dicts at all.) Since this introduces new complexity (involving updating only part of the ctf_pptrtab) which is only seen when a writable dict is in use, we introduce a new libctf-writable testsuite that contains lookup tests with no corresponding CTF-containing .c files (which can thus be run even on platforms with no .ctf-section support in the linker yet), and add a test to check that creation of pointers in children to types in parents and a following lookup by name works as expected. The non- writable case is tested in a new libctf-regression testsuite which is used to track now-fixed outright bugs in libctf. libctf/ChangeLog 2021-01-05 Nick Alcock <nick.alcock@oracle.com> * ctf-impl.h (ctf_dict_t) <ctf_pptrtab>: New. <ctf_pptrtab_len>: New. <ctf_pptrtab_typemax>: New. * ctf-create.c (ctf_serialize): Update accordingly. (ctf_add_reftype): Note that we don't need to update pptrtab here, despite updating ptrtab. * ctf-open.c (ctf_dict_close): Destroy the pptrtab. (ctf_import): Likewise. (ctf_import_unref): Likewise. * ctf-lookup.c (grow_pptrtab): New. (refresh_pptrtab): New, update a pptrtab. (ctf_lookup_by_name): Turn into a wrapper around (and rename to)... (ctf_lookup_by_name_internal): ... this: construct the pptrtab, and use it in addition to the parent's ptrtab when parent dicts are searched. * testsuite/libctf-regression/regression.exp: New testsuite for regression tests. * testsuite/libctf-regression/pptrtab: New test. testsuite/libctf-writable/writable.exp: New testsuite for tests of writable CTF dicts. * testsuite/libctf-writable/pptrtab*: New test.	6 years ago
Alan Modra	250d07de5c	Update year range in copyright notice of binutils files	6 years ago
Nick Alcock	53651de80f	libctf, include: support foreign-endianness symtabs with CTF The CTF symbol lookup machinery added recently has one deficit: it assumes the symtab is in the machine's native endianness. This is always true when the linker is writing out symtabs (because cross linkers byteswap symbols only after libctf has been called on them), but may be untrue in the cross case when the linker or another tool (objdump, etc) is reading them. Unfortunately the easy way to model this to the caller, as an endianness field in the ctf_sect_t, is precluded because doing so would change the size of the ctf_sect_t, which would be an ABI break. So, instead, allow the endianness of the symtab to be set after open time, by calling one of the two new API functions ctf_symsect_endianness (for ctf_dict_t's) or ctf_arc_symsect_endianness (for entire ctf_archive_t's). libctf calls these functions automatically for objects opened via any of the BFD-aware mechanisms (ctf_bfdopen, ctf_bfdopen_ctfsect, ctf_fdopen, ctf_open, or ctf_arc_open), but the various mechanisms that just take raw ctf_sect_t's will assume the symtab is in native endianness and need a later call to ctf_symsect_endianness to adjust it if needed. (This call is basically free if the endianness is actually native: it only costs anything if the symtab endianness was previously guessed wrong, and there is a symtab, and we are using it directly rather than using symtab indexing.) Obviously, calling ctf_lookup_by_symbol or ctf_symbol_next before the symtab endianness is correctly set will probably give wrong answers -- but you can set it at any time as long as it is before then. include/ChangeLog 2020-11-23 Nick Alcock <nick.alcock@oracle.com> ctf-api.h: Style nit: remove () on function names in comments. (ctf_sect_t): Mention endianness concerns. (ctf_symsect_endianness): New declaration. (ctf_arc_symsect_endianness): Likewise. libctf/ChangeLog 2020-11-23 Nick Alcock <nick.alcock@oracle.com> * ctf-impl.h (ctf_dict_t) <ctf_symtab_little_endian>: New. (struct ctf_archive_internal) <ctfi_symsect_little_endian>: Likewise. * ctf-create.c (ctf_serialize): Adjust for new field. * ctf-open.c (init_symtab): Note the semantics of repeated calls. (ctf_symsect_endianness): New. (ctf_bufopen_internal): Set ctf_symtab_little_endian suitably for the native endianness. (_Static_assert): Moved... (swap_thing): ... with this... * swap.h: ... to here. * ctf-util.c (ctf_elf32_to_link_sym): Use it, byteswapping the Elf32_Sym if the ctf_symtab_little_endian demands it. (ctf_elf64_to_link_sym): Likewise swap the Elf64_Sym if needed. * ctf-archive.c (ctf_arc_symsect_endianness): New, set the endianness of the symtab used by the dicts in an archive. (ctf_archive_iter_internal): Initialize to unknown (assumed native, do not call ctf_symsect_endianness). (ctf_dict_open_by_offset): Call ctf_symsect_endianness if need be. (ctf_dict_open_internal): Propagate the endianness down. (ctf_dict_open_sections): Likewise. * ctf-open-bfd.c (ctf_bfdopen_ctfsect): Get the endianness from the struct bfd and pass it down to the archive. * libctf.ver: Add ctf_symsect_endianness and ctf_arc_symsect_endianness.	6 years ago
Nick Alcock	97a2a623d0	libctf, include: add ctf_getsymsect and ctf_getstrsect libctf has long provided ctf_getdatasect, which hands back a pointer to the CTF section a (read-only) dict came from. But it has no such functions to return pointers to the ELF symbol table or string table it's working from, which is unfortunate because several libctf functions (ctf_open, ctf_fdopen, and ctf_bfdopen) figure out which string and symbol table to use themselves, and don't tell the user what they decided, so the caller can't agree on which symtab to use with libctf even if it wanted to. Add a pair of functions to return the symtab and strtab in use. Like ctf_getdatasect, these return ctf_sect_t structures by value, filled with all-NULL/0 content if a symtab or strtab is not being used. include/ChangeLog 2020-11-20 Nick Alcock <nick.alcock@oracle.com> * ctf-api.h (ctf_getsymsect): New. (ctf_getstrsect): Likewise. libctf/ChangeLog 2020-11-20 Nick Alcock <nick.alcock@oracle.com> * ctf-open.c (ctf_getsymsect): New. (ctf_getstrsect): Likewise. * libctf.ver: Add them.	6 years ago
Nick Alcock	1136c37971	libctf: symbol type linking support This adds facilities to write out the function info and data object sections, which efficiently map from entries in the symbol table to types. The write-side code is entirely new: the read-side code was merely significantly changed and support for indexed tables added (pointed to by the no-longer-unused cth_objtidxoff and cth_funcidxoff header fields). With this in place, you can use ctf_lookup_by_symbol to look up the types of symbols of function and object type (and, as before, you can use ctf_lookup_variable to look up types of file-scope variables not present in the symbol table, as long as you know their name: but variables that are also data objects are now found in the data object section instead.) (Compatible) file format change: The CTF spec has always said that the function info section looks much like the CTF_K_FUNCTIONs in the type section: an info word (including an argument count) followed by a return type and N argument types. This format is suboptimal: it means function symbols cannot be deduplicated and it causes a lot of ugly code duplication in libctf. But conveniently the compiler has never emitted this! Because it has always emitted a rather different format that libctf has never accepted, we can be sure that there are no instances of this function info section in the wild, and can freely change its format without compatibility concerns or a file format version bump. (And since it has never been emitted in any code that generated any older file format version, either, we need keep no code to read the format as specified at all!) So the function info section is now specified as an array of uint32_t, exactly like the object data section: each entry is a type ID in the type section which must be of kind CTF_K_FUNCTION, the prototype of this function. This allows function types to be deduplicated and also correctly encodes the fact that all functions declared in C really are types available to the program: so they should be stored in the type section like all other types. (In format v4, we will be able to represent the types of static functions as well, but that really does require a file format change.) We introduce a new header flag, CTF_F_NEWFUNCINFO, which is set if the new function info format is in use. A sufficiently new compiler will always set this flag. New libctf will always set this flag: old libctf will refuse to open any CTF dicts that have this flag set. If the flag is not set on a dict being read in, new libctf will disregard the function info section. Format v4 will remove this flag (or, rather, the flag has no meaning there and the bit position may be recycled for some other purpose). New API: Symbol addition: ctf_add_func_sym: Add a symbol with a given name and type. The type must be of kind CTF_K_FUNCTION (a function pointer). Internally this adds a name -> type mapping to the ctf_funchash in the ctf_dict. ctf_add_objt_sym: Add a symbol with a given name and type. The type kind can be anything, including function pointers. This adds to ctf_objthash. These both treat symbols as name -> type mappings: the linker associates symbol names with symbol indexes via the ctf_link_shuffle_syms callback, which sets up the ctf_dynsyms/ctf_dynsymidx/ctf_dynsymmax fields in the ctf_dict. Repeated relinks can add more symbols. Variables that are also exposed as symbols are removed from the variable section at serialization time. CTF symbol type sections which have enough pads, defined by CTF_INDEX_PAD_THRESHOLD (whether because they are in dicts with symbols where most types are unknown, or in archive where most types are defined in some child or parent dict, not in this specific dict) are sorted by name rather than symidx and accompanied by an index which associates each symbol type entry with a name: the existing ctf_lookup_by_symbol will map symbol indexes to symbol names and look the names up in the index automatically. (This is currently ELF-symbol-table-dependent, but there is almost nothing specific to ELF in here and we can add support for other symbol table formats easily). The compiler also uses index sections to communicate the contents of object file symbol tables without relying on any specific ordering of symbols: it doesn't need to sort them, and libctf will detect an unsorted index section via the absence of the new CTF_F_IDXSORTED header flag, and sort it if needed. Iteration: ctf_symbol_next: Iterator which returns the types and names of symbols one by one, either for function or data symbols. This does not require any sorting: the ctf_link machinery uses it to pull in all the compiler-provided symbols cheaply, but it is not restricted to that use. (Compatible) changes in API: ctf_lookup_by_symbol: can now be called for object and function symbols: never returns ECTF_NOTDATA (which is now not thrown by anything, but is kept for compatibility and because it is a plausible error that we might start throwing again at some later date). Internally we also have changes to the ctf-string functionality so that "external" strings (those where we track a string -> offset mapping, but only write out an offset) can be consulted via the usual means (ctf_strptr) before the strtab is written out. This is important because ctf_link_add_linker_symbol can now be handed symbols named via strtab offsets, and ctf_link_shuffle_syms must figure out their actual names by looking in the external symtab we have just been fed by the ctf_link_add_strtab callback, long before that strtab is written out. include/ChangeLog 2020-11-20 Nick Alcock <nick.alcock@oracle.com> * ctf-api.h (ctf_symbol_next): New. (ctf_add_objt_sym): Likewise. (ctf_add_func_sym): Likewise. * ctf.h: Document new function info section format. (CTF_F_NEWFUNCINFO): New. (CTF_F_IDXSORTED): New. (CTF_F_MAX): Adjust accordingly. libctf/ChangeLog 2020-11-20 Nick Alcock <nick.alcock@oracle.com> * ctf-impl.h (CTF_INDEX_PAD_THRESHOLD): New. (_libctf_nonnull_): Likewise. (ctf_in_flight_dynsym_t): New. (ctf_dict_t) <ctf_funcidx_names>: Likewise. <ctf_objtidx_names>: Likewise. <ctf_nfuncidx>: Likewise. <ctf_nobjtidx>: Likewise. <ctf_funcidx_sxlate>: Likewise. <ctf_objtidx_sxlate>: Likewise. <ctf_objthash>: Likewise. <ctf_funchash>: Likewise. <ctf_dynsyms>: Likewise. <ctf_dynsymidx>: Likewise. <ctf_dynsymmax>: Likewise. <ctf_in_flight_dynsym>: Likewise. (struct ctf_next) <u.ctn_next>: Likewise. (ctf_symtab_skippable): New prototype. (ctf_add_funcobjt_sym): Likewise. (ctf_dynhash_sort_by_name): Likewise. (ctf_sym_to_elf64): Rename to... (ctf_elf32_to_link_sym): ... this, and... (ctf_elf64_to_link_sym): ... this. * ctf-open.c (init_symtab): Check for lack of CTF_F_NEWFUNCINFO flag, and presence of index sections. Refactor out ctf_symtab_skippable and ctf_elf_to_link_sym, and use them. Use ctf_link_sym_t, not Elf64_Sym. Skip initializing objt or func sxlate sections if corresponding index section is present. Adjust for new func info section format. (ctf_bufopen_internal): Add ctf_err_warn to corrupt-file error handling. Report incorrect-length index sections. Always do an init_symtab, even if there is no symtab section (there may be index sections still). (flip_objts): Adjust comment: func and objt sections are actually identical in structure now, no need to caveat. (ctf_dict_close): Free newly-added data structures. ctf-create.c (ctf_create): Initialize them. (ctf_symtab_skippable): New, refactored out of init_symtab, with st_nameidx_set check added. (ctf_add_funcobjt_sym): New, add a function or object symbol to the ctf_objthash or ctf_funchash, by name. (ctf_add_objt_sym): Call it. (ctf_add_func_sym): Likewise. (symtypetab_delete_nonstatic_vars): New, delete vars also present as data objects. (CTF_SYMTYPETAB_EMIT_FUNCTION): New flag to symtypetab emitters: this is a function emission, not a data object emission. (CTF_SYMTYPETAB_EMIT_PAD): New flag to symtypetab emitters: emit pads for symbols with no type (only set for unindexed sections). (CTF_SYMTYPETAB_FORCE_INDEXED): New flag to symtypetab emitters: always emit indexed. (symtypetab_density): New, figure out section sizes. (emit_symtypetab): New, emit a symtypetab. (emit_symtypetab_index): New, emit a symtypetab index. (ctf_serialize): Call them, emitting suitably sorted symtypetab sections and indexes. Set suitable header flags. Copy over new fields. * ctf-hash.c (ctf_dynhash_sort_by_name): New, used to impose an order on symtypetab index sections. * ctf-link.c (ctf_add_type_mapping): Delete erroneous comment relating to code that was never committed. (ctf_link_one_variable): Improve variable name. (check_sym): New, symtypetab analogue of check_variable. (ctf_link_deduplicating_one_symtypetab): New. (ctf_link_deduplicating_syms): Likewise. (ctf_link_deduplicating): Call them. (ctf_link_deduplicating_per_cu): Note that we don't call them in this case (yet). (ctf_link_add_strtab): Set the error on the fp correctly. (ctf_link_add_linker_symbol): New (no longer a do-nothing stub), add a linker symbol to the in-flight list. (ctf_link_shuffle_syms): New (no longer a do-nothing stub), turn the in-flight list into a mapping we can use, now its names are resolvable in the external strtab. * ctf-string.c (ctf_str_rollback_atom): Don't roll back atoms with external strtab offsets. (ctf_str_rollback): Adjust comment. (ctf_str_write_strtab): Migrate ctf_syn_ext_strtab population from writeout time... (ctf_str_add_external): ... to string addition time. * ctf-lookup.c (ctf_lookup_var_key_t): Rename to... (ctf_lookup_idx_key_t): ... this, now we use it for syms too. <clik_names>: New member, a name table. (ctf_lookup_var): Adjust accordingly. (ctf_lookup_variable): Likewise. (ctf_lookup_by_id): Shuffle further up in the file. (ctf_symidx_sort_arg_cb): New, callback for... (sort_symidx_by_name): ... this new function to sort a symidx found to be unsorted (likely originating from the compiler). (ctf_symidx_sort): New, sort a symidx. (ctf_lookup_symbol_name): Support dynamic symbols with indexes provided by the linker. Use ctf_link_sym_t, not Elf64_Sym. Check the parent if a child lookup fails. (ctf_lookup_by_symbol): Likewise. Work for function symbols too. (ctf_symbol_next): New, iterate over symbols with types (without sorting). (ctf_lookup_idx_name): New, bsearch for symbol names in indexes. (ctf_try_lookup_indexed): New, attempt an indexed lookup. (ctf_func_info): Reimplement in terms of ctf_lookup_by_symbol. (ctf_func_args): Likewise. (ctf_get_dict): Move... * ctf-types.c (ctf_get_dict): ... here. * ctf-util.c (ctf_sym_to_elf64): Re-express as... (ctf_elf64_to_link_sym): ... this. Add new st_symidx field, and st_nameidx_set (always 0, so st_nameidx can be ignored). Look in the ELF strtab for names. (ctf_elf32_to_link_sym): Likewise, for Elf32_Sym. (ctf_next_destroy): Destroy ctf_next_t.u.ctn_next if need be. * libctf.ver: Add ctf_symbol_next, ctf_add_objt_sym and ctf_add_func_sym.	6 years ago
Nick Alcock	139633c307	libctf, include, binutils, gdb, ld: rename ctf_file_t to ctf_dict_t The naming of the ctf_file_t type in libctf is a historical curiosity. Back in the Solaris days, CTF dictionaries were originally generated as a separate file and then (sometimes) merged into objects: hence the datatype was named ctf_file_t, and known as a "CTF file". Nowadays, raw CTF is essentially never written to a file on its own, and the datatype changed name to a "CTF dictionary" years ago. So the term "CTF file" refers to something that is never a file! This is at best confusing. The type has also historically been known as a 'CTF container", which is even more confusing now that we have CTF archives which are also a sort of container (they contain CTF dictionaries), but which are never referred to as containers in the source code. So fix this by completing the renaming, renaming ctf_file_t to ctf_dict_t throughout, and renaming those few functions that refer to CTF files by name (keeping compatibility aliases) to refer to dicts instead. Old users who still refer to ctf_file_t will see (harmless) pointer-compatibility warnings at compile time, but the ABI is unchanged (since C doesn't mangle names, and ctf_file_t was always an opaque type) and things will still compile fine as long as -Werror is not specified. All references to CTF containers and CTF files in the source code are fixed to refer to CTF dicts instead. Further (smaller) renamings of annoyingly-named functions to come, as part of the process of souping up queries across whole archives at once (needed for the function info and data object sections). binutils/ChangeLog 2020-11-20 Nick Alcock <nick.alcock@oracle.com> * objdump.c (dump_ctf_errs): Rename ctf_file_t to ctf_dict_t. (dump_ctf_archive_member): Likewise. (dump_ctf): Likewise. Use ctf_dict_close, not ctf_file_close. * readelf.c (dump_ctf_errs): Rename ctf_file_t to ctf_dict_t. (dump_ctf_archive_member): Likewise. (dump_section_as_ctf): Likewise. Use ctf_dict_close, not ctf_file_close. gdb/ChangeLog 2020-11-20 Nick Alcock <nick.alcock@oracle.com> * ctfread.c: Change uses of ctf_file_t to ctf_dict_t. (ctf_fp_info::~ctf_fp_info): Call ctf_dict_close, not ctf_file_close. include/ChangeLog 2020-11-20 Nick Alcock <nick.alcock@oracle.com> * ctf-api.h (ctf_file_t): Rename to... (ctf_dict_t): ... this. Keep ctf_file_t around for compatibility. (struct ctf_file): Likewise rename to... (struct ctf_dict): ... this. (ctf_file_close): Rename to... (ctf_dict_close): ... this, keeping compatibility function. (ctf_parent_file): Rename to... (ctf_parent_dict): ... this, keeping compatibility function. All callers adjusted. * ctf.h: Rename references to ctf_file_t to ctf_dict_t. (struct ctf_archive) <ctfa_nfiles>: Rename to... <ctfa_ndicts>: ... this. ld/ChangeLog 2020-11-20 Nick Alcock <nick.alcock@oracle.com> * ldlang.c (ctf_output): This is a ctf_dict_t now. (lang_ctf_errs_warnings): Rename ctf_file_t to ctf_dict_t. (ldlang_open_ctf): Adjust comment. (lang_merge_ctf): Use ctf_dict_close, not ctf_file_close. * ldelfgen.h (ldelf_examine_strtab_for_ctf): Rename ctf_file_t to ctf_dict_t. Change opaque declaration accordingly. * ldelfgen.c (ldelf_examine_strtab_for_ctf): Adjust. * ldemul.h (examine_strtab_for_ctf): Likewise. (ldemul_examine_strtab_for_ctf): Likewise. * ldeuml.c (ldemul_examine_strtab_for_ctf): Likewise. libctf/ChangeLog 2020-11-20 Nick Alcock <nick.alcock@oracle.com> * ctf-impl.h: Rename ctf_file_t to ctf_dict_t: all declarations adjusted. (ctf_fileops): Rename to... (ctf_dictops): ... this. (ctf_dedup_t) <cd_id_to_file_t>: Rename to... <cd_id_to_dict_t>: ... this. (ctf_file_t): Fix outdated comment. <ctf_fileops>: Rename to... <ctf_dictops>: ... this. (struct ctf_archive_internal) <ctfi_file>: Rename to... <ctfi_dict>: ... this. * ctf-archive.c: Rename ctf_file_t to ctf_dict_t. Rename ctf_archive.ctfa_nfiles to ctfa_ndicts. Rename ctf_file_close to ctf_dict_close. All users adjusted. * ctf-create.c: Likewise. Refer to CTF dicts, not CTF containers. (ctf_bundle_t) <ctb_file>: Rename to... <ctb_dict): ... this. * ctf-decl.c: Rename ctf_file_t to ctf_dict_t. * ctf-dedup.c: Likewise. Rename ctf_file_close to ctf_dict_close. Refer to CTF dicts, not CTF containers. * ctf-dump.c: Likewise. * ctf-error.c: Likewise. * ctf-hash.c: Likewise. * ctf-inlines.h: Likewise. * ctf-labels.c: Likewise. * ctf-link.c: Likewise. * ctf-lookup.c: Likewise. * ctf-open-bfd.c: Likewise. * ctf-string.c: Likewise. * ctf-subr.c: Likewise. * ctf-types.c: Likewise. * ctf-util.c: Likewise. * ctf-open.c: Likewise. (ctf_file_close): Rename to... (ctf_dict_close): ...this. (ctf_file_close): New trivial wrapper around ctf_dict_close, for compatibility. (ctf_parent_file): Rename to... (ctf_parent_dict): ... this. (ctf_parent_file): New trivial wrapper around ctf_parent_dict, for compatibility. * libctf.ver: Add ctf_dict_close and ctf_parent_dict.	6 years ago
Nick Alcock	926c9e7665	libctf, binutils, include, ld: gettextize and improve error handling This commit follows on from the earlier commit "libctf, ld, binutils: add textual error/warning reporting for libctf" and converts every error in libctf that was reported using ctf_dprintf to use ctf_err_warn instead, gettextizing them in the process, using N_() where necessary to avoid doing gettext calls unless an error message is actually generated, and rephrasing some error messages for ease of translation. This requires a slight change in the ctf_errwarning_next API: this API is public but has not been in a release yet, so can still change freely. The problem is that many errors are emitted at open time (whether opening of a CTF dict, or opening of a CTF archive): the former of these throws away its incompletely-initialized ctf_file_t rather than return it, and the latter has no ctf_file_t at all. So errors and warnings emitted at open time cannot be stored in the ctf_file_t, and have to go elsewhere. We put them in a static local in ctf-subr.c (which is not very thread-safe: a later commit will improve things here): ctf_err_warn with a NULL fp adds to this list, and the public interface ctf_errwarning_next with a NULL fp retrieves from it. We need a slight exception from the usual iterator rules in this case: with a NULL fp, there is nowhere to store the ECTF_NEXT_END "error" which signifies the end of iteration, so we add a new err parameter to ctf_errwarning_next which is used to report such iteration-related errors. (If an fp is provided -- i.e., if not reporting open errors -- this is optional, but even if it's optional it's still an API change. This is actually useful from a usability POV as well, since ctf_errwarning_next is usually called when there's been an error, so overwriting the error code with ECTF_NEXT_END is not very helpful! So, unusually, ctf_errwarning_next now uses the passed fp for its error code only if no errp pointer is passed in, and leaves it untouched otherwise.) ld, objdump and readelf are adapted to call ctf_errwarning_next with a NULL fp to report open errors where appropriate. The ctf_err_warn API also has to change, gaining a new error-number parameter which is used to add the error message corresponding to that error number into the debug stream when LIBCTF_DEBUG is enabled: changing this API is easy at this point since we are already touching all existing calls to gettextize them. We need this because the debug stream should contain the errno's message, but the error reported in the error/warning stream should not, because the caller will probably report it themselves at failure time regardless, and reporting it in every error message that leads up to it leads to a ridiculous chattering on failure, which is likely to end up as ridiculous chattering on stderr (trimmed a bit): CTF error: `ld/testsuite/ld-ctf/A.c (0): lookup failure for type 3: flags 1: The parent CTF dictionary is unavailable' CTF error: `ld/testsuite/ld-ctf/A.c (0): struct/union member type hashing error during type hashing for type 80000001, kind 6: The parent CTF dictionary is unavailable' CTF error: `deduplicating link variable emission failed for ld/testsuite/ld-ctf/A.c: The parent CTF dictionary is unavailable' ld/.libs/lt-ld-new: warning: CTF linking failed; output will have no CTF section: `The parent CTF dictionary is unavailable' We only need to be told that the parent CTF dictionary is unavailable once, not over and over again! errmsgs are still emitted on warning generation, because warnings do not usually lead to a failure propagated up to the caller and reported there. Debug-stream messages are not translated. If translation is turned on, there will be a mixture of English and translated messages in the debug stream, but rather that than burden the translators with debug-only output. binutils/ChangeLog 2020-08-27 Nick Alcock <nick.alcock@oracle.com> * objdump.c (dump_ctf_archive_member): Move error- reporting... (dump_ctf_errs): ... into this separate function. (dump_ctf): Call it on open errors. * readelf.c (dump_ctf_archive_member): Move error- reporting... (dump_ctf_errs): ... into this separate function. Support calls with NULL fp. Adjust for new err parameter to ctf_errwarning_next. (dump_section_as_ctf): Call it on open errors. include/ChangeLog 2020-08-27 Nick Alcock <nick.alcock@oracle.com> * ctf-api.h (ctf_errwarning_next): New err parameter. ld/ChangeLog 2020-08-27 Nick Alcock <nick.alcock@oracle.com> * ldlang.c (lang_ctf_errs_warnings): Support calls with NULL fp. Adjust for new err parameter to ctf_errwarning_next. Only check for assertion failures when fp is non-NULL. (ldlang_open_ctf): Call it on open errors. * testsuite/ld-ctf/ctf.exp: Always use the C locale to avoid breaking the diags tests. libctf/ChangeLog 2020-08-27 Nick Alcock <nick.alcock@oracle.com> * ctf-subr.c (open_errors): New list. (ctf_err_warn): Calls with NULL fp append to open_errors. Add err parameter, and use it to decorate the debug stream with errmsgs. (ctf_err_warn_to_open): Splice errors from a CTF dict into the open_errors. (ctf_errwarning_next): Calls with NULL fp report from open_errors. New err param to report iteration errors (including end-of-iteration) when fp is NULL. (ctf_assert_fail_internal): Adjust ctf_err_warn call for new err parameter: gettextize. * ctf-impl.h (ctfo_get_vbytes): Add ctf_file_t parameter. (LCTF_VBYTES): Adjust. (ctf_err_warn_to_open): New. (ctf_err_warn): Adjust. (ctf_bundle): Used in only one place: move... * ctf-create.c: ... here. (enumcmp): Use ctf_err_warn, not ctf_dprintf, passing the err number down as needed. Don't emit the errmsg. Gettextize. (membcmp): Likewise. (ctf_add_type_internal): Likewise. (ctf_write_mem): Likewise. (ctf_compress_write): Likewise. Report errors writing the header or body. (ctf_write): Likewise. * ctf-archive.c (ctf_arc_write_fd): Use ctf_err_warn, not ctf_dprintf, and gettextize, as above. (ctf_arc_write): Likewise. (ctf_arc_bufopen): Likewise. (ctf_arc_open_internal): Likewise. * ctf-labels.c (ctf_label_iter): Likewise. * ctf-open-bfd.c (ctf_bfdclose): Likewise. (ctf_bfdopen): Likewise. (ctf_bfdopen_ctfsect): Likewise. (ctf_fdopen): Likewise. * ctf-string.c (ctf_str_write_strtab): Likewise. * ctf-types.c (ctf_type_resolve): Likewise. * ctf-open.c (get_vbytes_common): Likewise. Pass down the ctf dict. (get_vbytes_v1): Pass down the ctf dict. (get_vbytes_v2): Likewise. (flip_ctf): Likewise. (flip_types): Likewise. Use ctf_err_warn, not ctf_dprintf, and gettextize, as above. (upgrade_types_v1): Adjust calls. (init_types): Use ctf_err_warn, not ctf_dprintf, as above. (ctf_bufopen_internal): Likewise. Adjust calls. Transplant errors emitted into individual dicts into the open errors if this turns out to be a failed open in the end. * ctf-dump.c (ctf_dump_format_type): Adjust ctf_err_warn for new err argument. Gettextize. Don't emit the errmsg. (ctf_dump_funcs): Likewise. Collapse err label into its only case. (ctf_dump_type): Likewise. * ctf-link.c (ctf_create_per_cu): Adjust ctf_err_warn for new err argument. Gettextize. Don't emit the errmsg. (ctf_link_one_type): Likewise. (ctf_link_lazy_open): Likewise. (ctf_link_one_input_archive): Likewise. (ctf_link_deduplicating_count_inputs): Likewise. (ctf_link_deduplicating_open_inputs): Likewise. (ctf_link_deduplicating_close_inputs): Likewise. (ctf_link_deduplicating): Likewise. (ctf_link): Likewise. (ctf_link_deduplicating_per_cu): Likewise. Add some missed ctf_set_errnos to obscure error cases. * ctf-dedup.c (ctf_dedup_rhash_type): Adjust ctf_err_warn for new err argument. Gettextize. Don't emit the errmsg. (ctf_dedup_populate_mappings): Likewise. (ctf_dedup_detect_name_ambiguity): Likewise. (ctf_dedup_init): Likewise. (ctf_dedup_multiple_input_dicts): Likewise. (ctf_dedup_conflictify_unshared): Likewise. (ctf_dedup): Likewise. (ctf_dedup_rwalk_one_output_mapping): Likewise. (ctf_dedup_id_to_target): Likewise. (ctf_dedup_emit_type): Likewise. (ctf_dedup_emit_struct_members): Likewise. (ctf_dedup_populate_type_mapping): Likewise. (ctf_dedup_populate_type_mappings): Likewise. (ctf_dedup_emit): Likewise. (ctf_dedup_hash_type): Likewise. Fix a bit of messed-up error status setting. (ctf_dedup_rwalk_one_output_mapping): Likewise. Don't hide unknown-type-kind messages (which signify file corruption).	6 years ago
Nick Alcock	0f0c11f7fc	libctf, dedup: add deduplicator This adds the core deduplicator that the ctf_link machinery calls (possibly repeatedly) to link the CTF sections: it takes an array of input ctf_file_t's and another array that indicates which entries in the input array are parents of which other entries, and returns an array of outputs. The first output is always the ctf_file_t on which ctf_link/ctf_dedup/etc was called: the other outputs are child dicts that have the first output as their parent. include/ * ctf-api.h (CTF_LINK_SHARE_DUPLICATED): No longer unimplemented. libctf/ * ctf-impl.h (ctf_type_id_key): New, the key in the cd_id_to_file_t. (ctf_dedup): New, core deduplicator state. (ctf_file_t) <ctf_dedup>: New. <ctf_dedup_atoms>: New. <ctf_dedup_atoms_alloc>: New. (ctf_hash_type_id_key): New prototype. (ctf_hash_eq_type_id_key): Likewise. (ctf_dedup_atoms_init): Likewise. * ctf-hash.c (ctf_hash_eq_type_id_key): New. (ctf_dedup_atoms_init): Likewise. * ctf-create.c (ctf_serialize): Adjusted. (ctf_add_encoded): No longer static. (ctf_add_reftype): Likewise. * ctf-open.c (ctf_file_close): Destroy the ctf_dedup_atoms_alloc. * ctf-dedup.c: New file. * ctf-decls.h [!HAVE_DECL_STPCPY]: Add prototype. * configure.ac: Check for stpcpy. * Makefile.am: Add it. * Makefile.in: Regenerate. * config.h.in: Regenerate. * configure: Regenerate.	6 years ago
Nick Alcock	5f54462c6a	libctf, link: redo cu-mapping handling Now a bunch of stuff that doesn't apply to ld or any normal use of libctf, piled into one commit so that it's easier to ignore. The cu-mapping machinery associates incoming compilation unit names with outgoing names of CTF dictionaries that should correspond to them, for non-gdb CTF consumers that would like to group multiple TUs into a single child dict if conflicting types are found in it (the existing use case is one kernel module, one child CTF dict, even if the kernel module is composed of multiple CUs). The upcoming deduplicator needs to track not only the mapping from incoming CU name to outgoing dict name, but the inverse mapping from outgoing dict name to incoming CU name, so it can work over every CTF dict we might see in the output and link into it. So rejig the ctf-link machinery to do that. Simultaneously (because they are closely associated and were written at the same time), we add a new CTF_LINK_EMPTY_CU_MAPPINGS flag to ctf_link, which tells the ctf_link machinery to create empty child dicts for each outgoing CU mapping even if no CUs that correspond to it exist in the link. This is a bit (OK, quite a lot) of a waste of space, but some existing consumers require it. (Nobody else should use it.) Its value is not consecutive with existing CTF_LINK flag values because we're about to add more flags that are conceptually closer to the existing ones than this one is. include/ * ctf-api.h (CTF_LINK_EMPTY_CU_MAPPINGS): New. libctf/ * ctf-impl.h (ctf_file_t): Improve comments. <ctf_link_cu_mapping>: Split into... <ctf_link_in_cu_mapping>: ... this... <ctf_link_out_cu_mapping>: ... and this. * ctf-create.c (ctf_serialize): Adjust. * ctf-open.c (ctf_file_close): Likewise. * ctf-link.c (ctf_create_per_cu): Look things up in the in_cu_mapping instead of the cu_mapping. (ctf_link_add_cu_mapping): The deduplicating link will define what happens if many FROMs share a TO. (ctf_link_add_cu_mapping): Create in_cu_mapping and out_cu_mapping. Do not create ctf_link_outputs here any more, or create per-CU dicts here: they are already created when needed. (ctf_link_one_variable): Log a debug message if we skip a variable due to its type being concealed in a CU-mapped link. (This is probably too common a case to make into a warning.) (ctf_link): Create empty per-CU dicts if requested.	6 years ago
Nick Alcock	1fa7a0c24e	libctf: sort out potential refcount loops When you link TUs that contain conflicting types together, the resulting CTF section is an archive containing many CTF dicts. These dicts appear in ctf_link_outputs of the shared dict, with each ctf_import'ing that shared dict. ctf_importing a dict bumps its refcount to stop it going away while it's in use -- but if the shared dict (whose refcount is bumped) has the child dict (doing the bumping) in its ctf_link_outputs, we have a refcount loop, since the child dict only un-ctf_imports and drops the parent's refcount when it is freed, but the child is only freed when the parent's refcount falls to zero. (In the future, this will be able to go wrong on the inputs too, when an ld -r'ed deduplicated output with conflicts is relinked. Right now this cannot happen because we don't ctf_import such dicts at all. This will be fixed in a later commit in this series.) Fix this by introducing an internal-use-only ctf_import_unref function that imports a parent dict witthout bumping the parent's refcount, and using it when we create per-CU outputs. This function is only safe to use if you know the parent cannot go away while the child exists: but if the parent owns the child, as here, this is necessarily true. Record in the ctf_file_t whether a parent was imported via ctf_import or ctf_import_unref, so that if you do another ctf_import later on (or a ctf_import_unref) it can decide whether to drop the refcount of the existing parent being replaced depending on which function you used to import that one. Adjust ctf_serialize so that rather than doing a ctf_import (which is wrong if the original import was ctf_import_unref'fed), we just copy the parent field and refcount over and forcibly flip the unref flag on on the old copy we are going to discard. ctf_file_close also needs a bit of tweaking to only close the parent if it was not imported with ctf_import_unref: while we're at it, guard against repeated closes with a refcount of zero and stop them causing double-frees, even if destruction of things freed inside ctf_file_close cause such recursion. Verified no leaks or accesses to freed memory after all of this with valgrind. (It was leak-happy before.) libctf/ * ctf-impl.c (ctf_file_t) <ctf_parent_unreffed>: New. (ctf_import_unref): New. * ctf-open.c (ctf_file_close) Drop the refcount all the way to zero. Don't recurse back in if the refcount is already zero. (ctf_import): Check ctf_parent_unreffed before deciding whether to close a pre-existing parent. Set it to zero. (ctf_import_unreffed): New, as above, setting ctf_parent_unreffed to 1. * ctf-create.c (ctf_serialize): Do not ctf_import into the new child: use direct assignment, and set unreffed on the new and old children. * ctf-link.c (ctf_create_per_cu): Import the parent using ctf_import_unreffed.	6 years ago
Nick Alcock	8b37e7b63e	libctf, ld, binutils: add textual error/warning reporting for libctf This commit adds a long-missing piece of infrastructure to libctf: the ability to report errors and warnings using all the power of printf, rather than being restricted to one errno value. Internally, libctf calls ctf_err_warn() to add errors and warnings to a list: a new iterator ctf_errwarning_next() then consumes this list one by one and hands it to the caller, which can free it. New errors and warnings are added until the list is consumed by the caller or the ctf_file_t is closed, so you can dump them at intervals. The caller can of course choose to print only those warnings it wants. (I am not sure whether we want objdump, readelf or ld to print warnings or not: right now I'm printing them, but maybe we only want to print errors? This entirely depends on whether warnings are voluminous things describing e.g. the inability to emit single types because of name clashes or something. There are no users of this infrastructure yet, so it's hard to say.) There is no internationalization here yet, but this at least adds a place where internationalization can be added, to one of ctf_errwarning_next or ctf_err_warn. We also provide a new ctf_assert() function which uses this infrastructure to provide non-fatal assertion failures while emitting an assert-like string to the caller: to save space and avoid needlessly duplicating unchanging strings, the assertion test is inlined but the print-things-out failure case is not. All assertions in libctf will be converted to use this machinery in future commits and propagate assertion-failure errors up, so that the linker in particular cannot be killed by libctf assertion failures when it could perfectly well just print warnings and drop the CTF section. include/ * ctf-api.h (ECTF_INTERNAL): Adjust error text. (ctf_errwarning_next): New. libctf/ * ctf-impl.h (ctf_assert): New. (ctf_err_warning_t): Likewise. (ctf_file_t) <ctf_errs_warnings>: Likewise. (ctf_err_warn): New prototype. (ctf_assert_fail_internal): Likewise. * ctf-inlines.h (ctf_assert_internal): Likewise. * ctf-open.c (ctf_file_close): Free ctf_errs_warnings. * ctf-create.c (ctf_serialize): Copy it on serialization. * ctf-subr.c (ctf_err_warn): New, add an error/warning. (ctf_errwarning_next): New iterator, free and pass back errors/warnings in succession. * libctf.ver (ctf_errwarning_next): Add. ld/ * ldlang.c (lang_ctf_errs_warnings): New, print CTF errors and warnings. Assert when libctf asserts. (lang_merge_ctf): Call it. (land_write_ctf): Likewise. binutils/ * objdump.c (ctf_archive_member): Print CTF errors and warnings. * readelf.c (dump_ctf_archive_member): Likewise.	6 years ago
Nick Alcock	ec388c16cd	libctf: error out on corrupt CTF with invalid header flags If corrupt CTF with invalid header flags is passed in, return the new error ECTF_FLAGS. include/ * ctf-api.h (ECTF_FLAGS): New. (ECTF_NERR): Adjust. * ctf.h (CTF_F_MAX): New. libctf/ * ctf-open.c (ctf_bufopen_internal): Diagnose invalid flags.	6 years ago
Nick Alcock	2399827bfa	libctf: add ctf_ref This allows you to bump the refcount on a ctf_file_t, so that you can smuggle it out of iterators which open and close the ctf_file_t for you around the loop body (like ctf_archive_iter). You still can't use this to preserve a ctf_file_t for longer than the lifetime of its containing entity (e.g. ctf_archive). include/ * ctf-api.h (ctf_ref): New. libctf/ * libctf.ver (ctf_ref): New. * ctf-open.c (ctf_ref): Implement it.	6 years ago
Nick Alcock	2484ca436a	libctf, open: drop unnecessary historical wart around forwards When opening, we consider a forward with a kind above the maximum allowable set of kinds and a forward of kind CTF_K_UNKNOWN to be a forward to a struct. Whatever CTF version it was that produced forwards with no associated kind, it predates anything we can read: remove this wart. libctf/ * ctf-open.c (init_types): Remove typeless CTF_K_FORWARD special-casing.	6 years ago
Nick Alcock	fe4c2d5563	libctf: create: non-root-visible types should not appear in name tables We were accidentally interning newly-added and newly-opened non-root-visible types into name tables, and removing names from name tables when such types were removed. This is very wrong: the whole point of non-root-visible types is they do not go in name tables and cannot be looked up by name. This bug made non-root-visible types basically identical to root-visible types, right back to the earliest days of libctf in the Solaris era. libctf/ * ctf-open.c (init_types): Only intern root-visible types. * ctf-create.c (ctf_dtd_insert): Likewise. (ctf_dtd_delete): Only remove root-visible types. (ctf_rollback): Likewise. (ctf_add_generic): Adjust. (ctf_add_struct_sized): Adjust comment. (ctf_add_union_sized): Likewise. (ctf_add_enum): Likewise. * ctf-impl.h (ctf_dtd_insert): Adjust prototype.	7 years ago
Alan Modra	b3adc24a07	Update year range in copyright notice of binutils files	7 years ago
Nick Alcock	fa56cdcd24	libctf: fix tabdamage A little tabdamage predating the linker patch series has crept in. New in v5. libctf/ * ctf-open.c (ctf_bufopen_internal): Fix tabdamage. * ctf-types.c (ctf_type_lname): Likewise.	7 years ago
Nick Alcock	ad613f1d06	libctf: fix refcount leak in ctf_import Calling ctf_import (fp, NULL) to cancel out a pre-existing import leaked the refcnt increment on the parent, so it could never be freed. New in v4. libctf/ * ctf-open.c (ctf_import): Do not leak a ctf_file_t ref on every ctf_import after the first for a given file.	7 years ago
Nick Alcock	de07e349be	libctf: remove ctf_malloc, ctf_free and ctf_strdup These just get in the way of auditing for erroneous usage of strdup and add a huge irregular surface of "ctf_malloc or malloc? ctf_free or free? ctf_strdup or strdup?" ctf_malloc and ctf_free usage has not reliably matched up for many years, if ever, making the whole game pointless. Go back to malloc, free, and strdup like everyone else: while we're at it, fix a bunch of places where we weren't properly checking for OOM. This changes the interface of ctf_cuname_set and ctf_parent_name_set, which could strdup but could not return errors (like ENOMEM). New in v4. include/ * ctf-api.h (ctf_cuname_set): Can now fail, returning int. (ctf_parent_name_set): Likewise. libctf/ * ctf-impl.h (ctf_alloc): Remove. (ctf_free): Likewise. (ctf_strdup): Likewise. * ctf-subr.c (ctf_alloc): Remove. (ctf_free): Likewise. * ctf-util.c (ctf_strdup): Remove. * ctf-create.c (ctf_serialize): Use malloc, not ctf_alloc; free, not ctf_free; strdup, not ctf_strdup. (ctf_dtd_delete): Likewise. (ctf_dvd_delete): Likewise. (ctf_add_generic): Likewise. (ctf_add_function): Likewise. (ctf_add_enumerator): Likewise. (ctf_add_member_offset): Likewise. (ctf_add_variable): Likewise. (membadd): Likewise. (ctf_compress_write): Likewise. (ctf_write_mem): Likewise. * ctf-decl.c (ctf_decl_push): Likewise. (ctf_decl_fini): Likewise. (ctf_decl_sprintf): Likewise. Check for OOM. * ctf-dump.c (ctf_dump_append): Use malloc, not ctf_alloc; free, not ctf_free; strdup, not ctf_strdup. (ctf_dump_free): Likewise. (ctf_dump): Likewise. * ctf-open.c (upgrade_types_v1): Likewise. (init_types): Likewise. (ctf_file_close): Likewise. (ctf_bufopen_internal): Likewise. Check for OOM. (ctf_parent_name_set): Likewise: report the OOM to the caller. (ctf_cuname_set): Likewise. (ctf_import): Likewise. * ctf-string.c (ctf_str_purge_atom_refs): Use malloc, not ctf_alloc; free, not ctf_free; strdup, not ctf_strdup. (ctf_str_free_atom): Likewise. (ctf_str_create_atoms): Likewise. (ctf_str_add_ref_internal): Likewise. (ctf_str_remove_ref): Likewise. (ctf_str_write_strtab): Likewise.	7 years ago
Nick Alcock	99dc3ebdff	libctf: properly handle ctf_add_type of forwards and self-reffing structs The code to handle structures (and unions) that refer to themselves in ctf_add_type is extremely dodgy. It works by looking through the list of not-yet-committed types for a structure with the same name as the structure in question and assuming, if it finds it, that this must be a reference to the same type. This is a linear search that gets ever slower as the dictionary grows, requiring you to call ctf_update at intervals to keep performance tolerable: but if you do that, you run into the problem that if a forward declared before the ctf_update is changed to a structure afterwards, ctf_update explodes. The last commit fixed most of this: this commit can use it, adding a new ctf_add_processing hash that tracks source type IDs that are currently being processed and uses it to avoid infinite recursion rather than the dynamic type list: we split ctf_add_type into a ctf_add_type_internal, so that ctf_add_type itself can become a wrapper that empties out this being-processed hash once the entire recursive type addition is over. Structure additions themselves avoid adding their dependent types quite so much by checking the type mapping and avoiding re-adding types we already know we have added. We also add support for adding forwards to dictionaries that already contain the thing they are a forward to: we just silently return the original type. v4: return existing struct/union/enum types properly, rather than using an uninitialized variable: shrinks sizes of CTF sections back down to roughly where they were in v1/v2 of this patch series. v5: fix tabdamage. libctf/ * ctf-impl.h (ctf_file_t) <ctf_add_processing>: New. * ctf-open.c (ctf_file_close): Free it. * ctf-create.c (ctf_serialize): Adjust. (membcmp): When reporting a conflict due to an error, report the error. (ctf_add_type): Turn into a ctf_add_processing wrapper. Rename to... (ctf_add_type_internal): ... this. Hand back types we are already in the middle of adding immediately. Hand back structs/unions with the same number of members immediately. Do not walk the dynamic list. Call ctf_add_type_internal, not ctf_add_type. Handle forwards promoted to other types and the inverse case identically. Add structs to the mapping as soon as we intern them, before they gain any members.	7 years ago
Nick Alcock	676c3ecbad	libctf: avoid the need to ever use ctf_update The method of operation of libctf when the dictionary is writable has before now been that types that are added land in the dynamic type section, which is a linked list and hash of IDs -> dynamic type definitions (and, recently a hash of names): the DTDs are a bit of CTF representing the ctf_type_t and ad hoc C structures representing the vlen. Historically, libctf was unable to do anything with these types, not even look them up by ID, let alone by name: if you wanted to do that say if you were adding a type that depended on one you just added) you called ctf_update, which serializes all the DTDs into a CTF file and reopens it, copying its guts over the fp it's called with. The ctf_updated types are then frozen in amber and unchangeable: all lookups will return the types in the static portion in preference to the dynamic portion, and we will refuse to re-add things that already exist in the static portion (and, of late, in the dynamic portion too). The libctf machinery remembers the boundary between static and dynamic types and looks in the right portion for each type. Lots of things still don't quite work with dynamic types (e.g. getting their size), but enough works to do a bunch of additions and then a ctf_update, most of the time. Except it doesn't, because ctf_add_type finds it necessary to walk the full dynamic type definition list looking for types with matching names, so it gets slower and slower with every type you add: fixing this requires calling ctf_update periodically for no other reason than to avoid massively slowing things down. This is all clunky and very slow but kind of works, until you consider that it is in fact possible and indeed necessary to modify one sort of type after it has been added: forwards. These are necessarily promoted to structs, unions or enums, and when they do so their type ID does not change. So all of a sudden we are changing types that already exist in the static portion. ctf_update gets massively confused by this and allocates space enough for the forward (with no members), but then emits the new dynamic type (with all the members) into it. You get an assertion failure after that, if you're lucky, or a coredump. So this commit rejigs things a bit and arranges to exclusively use the dynamic type definitions in writable dictionaries, and the static type definitions in readable dictionaries: we don't at any time have a mixture of static and dynamic types, and you don't need to call ctf_update to make things "appear". The ctf_dtbyname hash I introduced a few months ago, which maps things like "struct foo" to DTDs, is removed, replaced instead by a change of type of the four dictionaries which track names. Rather than just being (unresizable) ctf_hash_t's populated only at ctf_bufopen time, they are now a ctf_names_t structure, which is a pair of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used in readonly dictionaries, and the ctf_dynhash_t being used in writable ones. The decision as to which to use is centralized in the new functions ctf_lookup_by_rawname (which takes a type kind) and ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t .) This change lets us switch from using static to dynamic name hashes on the fly across the entirety of libctf without complexifying anything: in fact, because we now centralize the knowledge about how to map from type kind to name hash, it actually simplifies things and lets us throw out quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced by the dynamic half of the name tables), through to ctf_dtnextid (now that a dictionary's static portion is never referenced if the dictionary is writable, we can just use ctf_typemax to indicate the maximum type: dynamic or non-dynamic does not matter, and we no longer need to track the boundary between the types). You can now ctf_rollback() as far as you like, even past a ctf_update or for that matter a full writeout; all the iteration functions work just as well on writable as on read-only dictionaries; ctf_add_type no longer needs expensive duplicated code to run over the dynamic types hunting for ones it might be interested in; and the linker no longer needs a hack to call ctf_update so that calling ctf_add_type is not impossibly expensive. There is still a bit more complexity: some new code paths in ctf-types.c need to know how to extract information from dynamic types. This complexity will go away again in a few months when libctf acquires a proper intermediate representation. You can still call ctf_update if you like (it's public API, after all), but its only effect now is to set the point to which ctf_discard rolls back. Obviously something* still needs to serialize the CTF file before writeout, and this job is done by ctf_serialize, which does everything ctf_update used to except set the counter used by ctf_discard. It is automatically called by the various functions that do CTF writeout: nobody else ever needs to call it. With this in place, forwards that are promoted to non-forwards no longer crash the link, even if it happens tens of thousands of types later. v5: fix tabdamage. libctf/ * ctf-impl.h (ctf_names_t): New. (ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t. (ctf_file_t) <ctf_structs>: Likewise. <ctf_unions>: Likewise. <ctf_enums>: Likewise. <ctf_names>: Likewise. <ctf_lookups>: Improve comment. <ctf_ptrtab_len>: New. <ctf_prov_strtab>: New. <ctf_str_prov_offset>: New. <ctf_dtbyname>: Remove, redundant to the names hashes. <ctf_dtnextid>: Remove, redundant to ctf_typemax. (ctf_dtdef_t) <dtd_name>: Remove. <dtd_data>: Note that the ctt_name is now populated. (ctf_str_atom_t) <csa_offset>: This is now the strtab offset for internal strings too. <csa_external_offset>: New, the external strtab offset. (CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case. (ctf_name_table): New declaration. (ctf_lookup_by_rawname): Likewise. (ctf_lookup_by_rawhash): Likewise. (ctf_set_ctl_hashes): Likewise. (ctf_serialize): Likewise. (ctf_dtd_insert): Adjust. (ctf_simple_open_internal): Likewise. (ctf_bufopen_internal): Likewise. (ctf_list_empty_p): Likewise. (ctf_str_remove_ref): Likewise. (ctf_str_add): Returns uint32_t now. (ctf_str_add_ref): Likewise. (ctf_str_add_external): Now returns a boolean (int). * ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab for strings in the appropriate range. (ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM when adding the null string to the new strtab. (ctf_str_free_atoms): Destroy the ctf_prov_strtab. (ctf_str_add_ref_internal): Add make_provisional argument. If make_provisional, populate the offset and fill in the ctf_prov_strtab accordingly. (ctf_str_add): Return the offset, not the string. (ctf_str_add_ref): Likewise. (ctf_str_add_external): Return a success integer. (ctf_str_remove_ref): New, remove a single ref. (ctf_str_count_strtab): Do not count the initial null string's length or the existence or length of any unreferenced internal atoms. (ctf_str_populate_sorttab): Skip atoms with no refs. (ctf_str_write_strtab): Populate the nullstr earlier. Add one to the cts_len for the null string, since it is no longer done in ctf_str_count_strtab. Adjust for csa_external_offset rename. Populate the csa_offset for both internal and external cases. Flush the ctf_prov_strtab afterwards, and reset the ctf_str_prov_offset. * ctf-create.c (ctf_grow_ptrtab): New. (ctf_create): Call it. Initialize new fields rather than old ones. Tell ctf_bufopen_internal that this is a writable dictionary. Set the ctl hashes and data model. (ctf_update): Rename to... (ctf_serialize): ... this. Leave a compatibility function behind. Tell ctf_simple_open_internal that this is a writable dictionary. Pass the new fields along from the old dictionary. Drop ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name. Do not zero out the DTD's ctt_name. (ctf_prefixed_name): Rename to... (ctf_name_table): ... this. No longer return a prefixed name: return the applicable name table instead. (ctf_dtd_insert): Use it, and use the right name table. Pass in the kind we're adding. Migrate away from dtd_name. (ctf_dtd_delete): Adjust similarly. Remove the ref to the deleted ctt_name. (ctf_dtd_lookup_type_by_name): Remove. (ctf_dynamic_type): Always return NULL on read-only dictionaries. No longer check ctf_dtnextid: check ctf_typemax instead. (ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead. (ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use ctf_name_table and the right name table, and migrate away from dtd_name as in ctf_dtd_delete. (ctf_add_generic): Pass in the kind explicitly and pass it to ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away from dtd_name to using ctf_str_add_ref to populate the ctt_name. Grow the ptrtab if needed. (ctf_add_encoded): Pass in the kind. (ctf_add_slice): Likewise. (ctf_add_array): Likewise. (ctf_add_function): Likewise. (ctf_add_typedef): Likewise. (ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking ctt_name rather than dtd_name. (ctf_add_struct_sized): Pass in the kind. Use ctf_lookup_by_rawname, not ctf_hash_lookup_type / ctf_dtd_lookup_type_by_name. (ctf_add_union_sized): Likewise. (ctf_add_enum): Likewise. (ctf_add_enum_encoded): Likewise. (ctf_add_forward): Likewise. (ctf_add_type): Likewise. (ctf_compress_write): Call ctf_serialize: adjust for ctf_size not being initialized until after the call. (ctf_write_mem): Likewise. (ctf_write): Likewise. * ctf-archive.c (arc_write_one_ctf): Likewise. * ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not ctf_hash_lookup_type. (ctf_lookup_by_id): No longer check the readonly types if the dictionary is writable. * ctf-open.c (init_types): Assert that this dictionary is not writable. Adjust to use the new name hashes, ctf_name_table, and ctf_ptrtab_len. GNU style fix for the final ptrtab scan. (ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR if set. Drop out early when dictionary is writable. Split the ctf_lookups initialization into... (ctf_set_cth_hashes): ... this new function. (ctf_simple_open_internal): Adjust. New 'writable' parameter. (ctf_simple_open): Adjust accordingly. (ctf_bufopen): Likewise. (ctf_file_close): Destroy the appropriate name hashes. No longer destroy ctf_dtbyname, which is gone. (ctf_getdatasect): Remove spurious "extern". * ctf-types.c (ctf_lookup_by_rawname): New, look up types in the specified name table, given a kind. (ctf_lookup_by_rawhash): Likewise, given a ctf_names_t . (ctf_member_iter): Add support for iterating over the dynamic type list. (ctf_enum_iter): Likewise. (ctf_variable_iter): Likewise. (ctf_type_rvisit): Likewise. (ctf_member_info): Add support for types in the dynamic type list. (ctf_enum_name): Likewise. (ctf_enum_value): Likewise. (ctf_func_type_info): Likewise. (ctf_func_type_args): Likewise. ctf-link.c (ctf_accumulate_archive_names): No longer call ctf_update. (ctf_link_write): Likewise. (ctf_link_intern_extern_string): Adjust for new ctf_str_add_external return value. (ctf_link_add_strtab): Likewise. * ctf-util.c (ctf_list_empty_p): New.	7 years ago
Nick Alcock	5ae6af75b5	libctf: eschew C99 for loop initial declarations We shouldn't use these, since binutils doesn't require a C99-capable compiler yet. New in v3. v5: fix tabdamage. libctf/ * ctf-open.c (flip_lbls): Eschew for-loop initial declarations. (flip_objts): Likewise. (flip_vars): Likewise. (flip_types): Likewise.	7 years ago
Nick Alcock	49ea9b450b	libctf: add CU-mapping machinery Once the deduplicator is capable of actually detecting conflicting types with the same name (i.e., not yet) we will place such conflicting types, and types that depend on them, into CTF dictionaries that are the child of the main dictionary we usually emit: currently, this will lead to the .ctf section becoming a CTF archive rather than a single dictionary, with the default-named archive member (_CTF_SECTION, or NULL) being the main shared dictionary with most of the types in it. By default, the sections are named after the compilation unit they come from (complete path and all), with the cuname field in the CTF header providing further evidence of the name without requiring the caller to engage in tiresome parsing. But some callers may not wish the mapping from input CU to output sub-dictionary to be purely CU-based. The machinery here allows this to be freely changed, in two ways: - callers can call ctf_link_add_cu_mapping to specify that a single input compilation unit should have its types placed in some other CU if they conflict: the CU will always be created, even if empty, so the consuming program can depend on its existence. You can map multiple input CUs to one output CU to force all their types to be merged together: if some of those types conflict, the behaviour is currently unspecified (the new deduplicator will specify it). - callers can call ctf_link_set_memb_name_changer to provide a function which is passed every CTF sub-dictionary name in turn (including _CTF_SECTION) and can return a new name, or NULL if no change is desired. The mapping from input to output names should not map two input names to the same output name: if this happens, the two are not merged but will result in an archive with two members with the same name (technically valid, but it's hard to access the second same-named member: you have to do an iteration over archive members). This is used by the kernel's ctfarchive machinery (not yet upstream) to encode CTF under member names like {module name}.ctf rather than .ctf.CU, but it is anticipated that other large projects may wish to have their own storage for CTF outside of .ctf sections and may wish to have new naming schemes that suit their special-purpose consumers. New in v3. v4: check for strdup failure. v5: fix tabdamage. include/ * ctf-api.h (ctf_link_add_cu_mapping): New. (ctf_link_memb_name_changer_f): New. (ctf_link_set_memb_name_changer): New. libctf/ * ctf-impl.h (ctf_file_t) <ctf_link_cu_mappping>: New. <ctf_link_memb_name_changer>: Likewise. <ctf_link_memb_name_changer_arg>: Likewise. * ctf-create.c (ctf_update): Update accordingly. * ctf-open.c (ctf_file_close): Likewise. * ctf-link.c (ctf_create_per_cu): Apply the cu mapping. (ctf_link_add_cu_mapping): New. (ctf_link_set_memb_name_changer): Likewise. (ctf_change_parent_name): New. (ctf_name_list_accum_cb_arg_t) <dynames>: New, storage for names allocated by the caller's ctf_link_memb_name_changer. <ndynames>: Likewise. (ctf_accumulate_archive_names): Call the ctf_link_memb_name_changer. (ctf_link_write): Likewise (for _CTF_SECTION only): also call ctf_change_parent_name. Free any resulting names.	7 years ago
Nick Alcock	886453cbbc	libctf: map from old to corresponding newly-added types in ctf_add_type This lets you call ctf_type_mapping (dest_fp, src_fp, src_type_id) and get told what type ID the corresponding type has in the target ctf_file_t. This works even if it was added by a recursive call, and because it is stored in the target ctf_file_t it works even if we had to add one type to multiple ctf_file_t's as part of conflicting type handling. We empty out this mapping after every archive is linked: because it maps input to output fps, and we only visit each input fp once, its contents are rendered entirely useless every time the source fp changes. v3: add several missing mapping additions. Add ctf_dynhash_empty, and empty after every input archive. v5: fix tabdamage. libctf/ * ctf-impl.h (ctf_file_t): New field ctf_link_type_mapping. (struct ctf_link_type_mapping_key): New. (ctf_hash_type_mapping_key): Likewise. (ctf_hash_eq_type_mapping_key): Likewise. (ctf_add_type_mapping): Likewise. (ctf_type_mapping): Likewise. (ctf_dynhash_empty): Likewise. * ctf-open.c (ctf_file_close): Update accordingly. * ctf-create.c (ctf_update): Likewise. (ctf_add_type): Populate the mapping. * ctf-hash.c (ctf_hash_type_mapping_key): Hash a type mapping key. (ctf_hash_eq_type_mapping_key): Check the key for equality. (ctf_dynhash_insert): Fix comment typo. (ctf_dynhash_empty): New. * ctf-link.c (ctf_add_type_mapping): New. (ctf_type_mapping): Likewise. (empty_link_type_mapping): New. (ctf_link_one_input_archive): Call it.	7 years ago
Nick Alcock	72c83edd92	libctf: add the ctf_link machinery This is the start of work on the core of the linking mechanism for CTF sections. This commit handles the type and string sections. The linker calls these functions in sequence: ctf_link_add_ctf: to add each CTF section in the input in turn to a newly-created ctf_file_t (which will appear in the output, and which itself will become the shared parent that contains types that all TUs have in common (in all link modes) and all types that do not have conflicting definitions between types (by default). Input files that are themselves products of ld -r are supported, though this is not heavily tested yet. ctf_link: called once all input files are added to merge the types in all the input containers into the output container, eliminating duplicates. ctf_link_add_strtab: called once the ELF string table is finalized and all its offsets are known, this calls a callback provided by the linker which returns the string content and offset of every string in the ELF strtab in turn: all these strings which appear in the input CTF strtab are eliminated from it in favour of the ELF strtab: equally, any strings that only appear in the input strtab will reappear in the internal CTF strtab of the output. ctf_link_shuffle_syms (not yet implemented): called once the ELF symtab is finalized, this calls a callback provided by the linker which returns information on every symbol in turn as a ctf_link_sym_t. This is then used to shuffle the function info and data object sections in the CTF section into symbol table order, eliminating the index sections which map those sections to symbol names before that point. Currently just returns ECTF_NOTYET. ctf_link_write: Returns a buffer containing either a serialized ctf_file_t (if there are no types with conflicting definitions in the object files in the link) or a ctf_archive_t containing a large ctf_file_t (the common types) and a bunch of small ones named after individual CUs in which conflicting types are found (containing the conflicting types, and all types that reference them). A threshold size above which compression takes place is passed as one parameter. (Currently, only gzip compression is supported, but I hope to add lzma as well.) Lifetime rules for this are simple: don't close the input CTF files until you've called ctf_link for the last time. We do not assume that symbols or strings passed in by the callback outlast the call to ctf_link_add_strtab or ctf_link_shuffle_syms. Right now, the duplicate elimination mechanism is the one already present as part of the ctf_add_type function, and is not particularly good: it misses numerous actual duplicates, and the conflicting-types detection hardly ever reports that types conflict, even when they do (one of them just tends to get silently dropped): it is also very slow. This will all be fixed in the next few weeks, but the fix hardly touches any of this code, and the linker does work without it, just not as well as it otherwise might. (And when no CTF section is present, there is no effect on performance, of course. So only people using a trunk GCC with not-yet-committed patches will even notice. By the time it gets upstream, things should be better.) v3: Fix error handling. v4: check for strdup failure. v5: fix tabdamage. include/ * ctf-api.h (struct ctf_link_sym): New, a symbol in flight to the libctf linking machinery. (CTF_LINK_SHARE_UNCONFLICTED): New. (CTF_LINK_SHARE_DUPLICATED): New. (ECTF_LINKADDEDLATE): New, replacing ECTF_UNUSED. (ECTF_NOTYET): New, a 'not yet implemented' message. (ctf_link_add_ctf): New, add an input file's CTF to the link. (ctf_link): New, merge the type and string sections. (ctf_link_strtab_string_f): New, callback for feeding strtab info. (ctf_link_iter_symbol_f): New, callback for feeding symtab info. (ctf_link_add_strtab): New, tell the CTF linker about the ELF strtab's strings. (ctf_link_shuffle_syms): New, ask the CTF linker to shuffle its symbols into symtab order. (ctf_link_write): New, ask the CTF linker to write the CTF out. libctf/ * ctf-link.c: New file, linking of the string and type sections. * Makefile.am (libctf_a_SOURCES): Add it. * Makefile.in: Regenerate. * ctf-impl.h (ctf_file_t): New fields ctf_link_inputs, ctf_link_outputs. * ctf-create.c (ctf_update): Update accordingly. * ctf-open.c (ctf_file_close): Likewise. * ctf-error.c (_ctf_errlist): Updated with new errors.	7 years ago
Nick Alcock	d851ecd373	libctf: support getting strings from the ELF strtab The CTF file format has always supported "external strtabs", which internally are strtab offsets with their MSB on: such refs get their strings from the strtab passed in at CTF file open time: this is usually intended to be the ELF strtab, and that's what this implementation is meant to support, though in theory the external strtab could come from anywhere. This commit adds support for these external strings in the ctf-string.c strtab tracking layer. It's quite easy: we just add a field csa_offset to the atoms table that tracks all strings: this field tracks the offset of the string in the ELF strtab (with its MSB already on, courtesy of a new macro CTF_SET_STID), and adds a new function that sets the csa_offset to the specified offset (plus MSB). Then we just need to avoid writing out strings to the internal strtab if they have csa_offset set, and note that the internal strtab is shorter than it might otherwise be. (We could in theory save a little more time here by eschewing sorting such strings, since we never actually write the strings out anywhere, but that would mean storing them separately and it's just not worth the complexity cost until profiling shows it's worth doing.) We also have to go through a bit of extra effort at variable-sorting time. This was previously using direct references to the internal strtab: it couldn't use ctf_strptr or ctf_strraw because the new strtab is not yet ready to put in its usual field (in a ctf_file_t that hasn't even been allocated yet at this stage): but now we're using the external strtab, this will no longer do because it'll be looking things up in the wrong strtab, with disastrous results. Instead, pass the new internal strtab in to a new ctf_strraw_explicit function which is just like ctf_strraw except you can specify a ne winternal strtab to use. But even now that it is using a new internal strtab, this is not quite enough: it can't look up strings in the external strtab because ld hasn't written it out yet, and when it does will write it straight to disk. Instead, when we write the internal strtab, note all the offset -> string mappings that we have noted belong in the external strtab to a new "synthetic external strtab" dynhash, ctf_syn_ext_strtab, and look in there at ctf_strraw time if it is set. This uses minimal extra memory (because only strings in the external strtab that we actually use are stored, and even those come straight out of the atoms table), but let both variable sorting and name interning when ctf_bufopen is next called work fine. (This also means that we don't need to filter out spurious ECTF_STRTAB warnings from ctf_bufopen but can pass them back to the caller, once we wrap ctf_bufopen so that we have a new internal variant of ctf_bufopen etc that we can pass the synthetic external strtab to. That error has been filtered out since the days of Solaris libctf, which didn't try to handle the problem of getting external strtabs right at construction time at all.) v3: add the synthetic strtab and all associated machinery. v5: fix tabdamage. include/ * ctf.h (CTF_SET_STID): New. libctf/ * ctf-impl.h (ctf_str_atom_t) <csa_offset>: New field. (ctf_file_t) <ctf_syn_ext_strtab>: Likewise. (ctf_str_add_ref): Name the last arg. (ctf_str_add_external) New. (ctf_str_add_strraw_explicit): Likewise. (ctf_simple_open_internal): Likewise. (ctf_bufopen_internal): Likewise. * ctf-string.c (ctf_strraw_explicit): Split from... (ctf_strraw): ... here, with new support for ctf_syn_ext_strtab. (ctf_str_add_ref_internal): Return the atom, not the string. (ctf_str_add): Adjust accordingly. (ctf_str_add_ref): Likewise. Move up in the file. (ctf_str_add_external): New: update the csa_offset. (ctf_str_count_strtab): Only account for strings with no csa_offset in the internal strtab length. (ctf_str_write_strtab): If the csa_offset is set, update the string's refs without writing the string out, and update the ctf_syn_ext_strtab. Make OOM handling less ugly. * ctf-create.c (struct ctf_sort_var_arg_cb): New. (ctf_update): Handle failure to populate the strtab. Pass in the new ctf_sort_var arg. Adjust for ctf_syn_ext_strtab addition. Call ctf_simple_open_internal, not ctf_simple_open. (ctf_sort_var): Call ctf_strraw_explicit rather than looking up strings by hand. * ctf-hash.c (ctf_hash_insert_type): Likewise (but using ctf_strraw). Adjust to diagnose ECTF_STRTAB nonetheless. * ctf-open.c (init_types): No longer filter out ECTF_STRTAB. (ctf_file_close): Destroy the ctf_syn_ext_strtab. (ctf_simple_open): Rename to, and reimplement as a wrapper around... (ctf_simple_open_internal): ... this new function, which calls ctf_bufopen_internal. (ctf_bufopen): Rename to, and reimplement as a wrapper around... (ctf_bufopen_internal): ... this new function, which sets ctf_syn_ext_strtab.	7 years ago
Nick Alcock	2db912ba1a	libctf: add the object index and function index sections No code handles these yet, but our latest GCC patches are generating them, so we have to be ready for them or erroneously conclude that we have file corruption. (This simultaneously fixes a longstanding bug, concealed because nothing was generating anything in the object or function info sections, where the end of the section was being tested against the wrong thing: it would have walked over the entire contents of the variable section and treated them as part of the function info section. This had to change now anyway because the new sections have landed in between.) include/ * ctf.h: Add object index and function index sections. Describe them. Improve the description of the variable section and clarify the constraints on backward-pointing type nodes. (ctf_header): Add cth_objtidxoff, cth_funcidxoff. libctf/ * ctf-open.c (init_symtab): Check for overflow against the right section. (upgrade_header): Set cth_objtidxoff, cth_funcidxoff to zero-length. (upgrade_types_v1): Note that these sections are not checked. (flip_header): Endian-swap the header fields. (flip_ctf): Endian-swap the sections. (flip_objts): Update comment. (ctf_bufopen): Check header offsets and alignment for validity.	7 years ago
Nick Alcock	6d5944fca6	libctf, bfd: fix ctf_bfdopen_ctfsect opening symbol and string sections The code in ctf_bfdopen_ctfsect (which is the ultimate place where you end up if you use ctf_open to open a CTF file and pull in the ELF string and symbol tables) was written before it was possible to actually test it, since the linker was not written. Now it is, it turns out that the previous code was completely nonfunctional: it assumed that you could load the symbol table via bfd_section_from_elf_index (...,elf_onesymtab()) and the string table via bfd_section_from_elf_index on the sh_link. Unfortunately BFD loads neither of these sections in the conventional fashion it uses for most others: the symbol table is immediately converted into internal form (which is useless for our purposes, since we also have to work in the absence of BFD for readelf, etc) and the string table is loaded specially via bfd_elf_get_str_section which is private to bfd/elf.c. So make this function public, export it in elf-bfd.h, and use it from libctf, which does something similar to what bfd_elf_sym_name and bfd_elf_string_from_elf_section do. Similarly, load the symbol table manually using bfd_elf_get_elf_syms and throw away the internal form it generates for us (we never use it). BFD allocates the strtab for us via bfd_alloc, so we can leave BFD to deallocate it: we allocate the symbol table ourselves before calling bfd_elf_get_elf_syms, so we still have to free it. Also change the rules around what you are allowed to provide: It is useful to provide a string section but no symbol table, because CTF sections can legitimately have no function info or data object sections while relying on the ELF strtab for some of their strings. So allow that combination. v4: adjust to upstream changes. ctf_bfdopen_ctfsect's first parameter is potentially unused again (if BFD is not in use for this link due to not supporting an ELF target). v5: fix tabdamage. bfd/ * elf-bfd.h (bfd_elf_get_str_section): Add. * elf.c (bfd_elf_get_str_section): No longer static. libctf/ * ctf-open-bfd.c: Add <assert.h>. (ctf_bfdopen_ctfsect): Open string and symbol tables using techniques borrowed from bfd_elf_sym_name. (ctf_new_archive_internal): Improve comment. * ctf-archive.c (ctf_arc_close): Do not free the ctfi_strsect. * ctf-open.c (ctf_bufopen): Allow opening with a string section but no symbol section, but not vice versa.	7 years ago
Nick Alcock	9b32cba44d	libctf, binutils: dump the CTF header The CTF header has before now been thrown away too soon to be dumped using the ctf_dump() machinery used by objdump and readelf: instead, a kludge involving debugging-priority dumps of the header offsets on every open was used. Replace this with proper first-class dumping machinery just like everything else in the CTF file, and have objdump and readelf use it. (The dumper already had an enum value in ctf_sect_names_t for this purpose, waiting to be used.) v5: fix tabdamage. libctf/ * ctf-impl.h (ctf_file_t): New field ctf_openflags. * ctf-open.c (ctf_bufopen): Set it. No longer dump header offsets. * ctf-dump.c (dump_header): New function, dump the CTF header. (ctf_dump): Call it. (ctf_dump_header_strfield): New function. (ctf_dump_header_sectfield): Likewise. binutils/ * objdump.c (dump_ctf_archive_member): Dump the CTF header. * readelf.c (dump_section_as_ctf): Likewise.	7 years ago
Nick Alcock	fd55eae84d	libctf: allow the header to change between versions libctf supports dynamic upgrading of the type table as file format versions change, but before now has not supported changes to the CTF header. Doing this is complicated by the baroque storage method used: the CTF header is kept prepended to the rest of the CTF data, just as when read from the file, and written out from there, and is endian-flipped in place. This makes accessing it needlessly hard and makes it almost impossible to make the header larger if we add fields. The general storage machinery around the malloced ctf pointer (the 'ctf_base') is also overcomplicated: the pointer is sometimes malloced locally and sometimes assigned from a parameter, so freeing it requires checking to see if that parameter was used, needlessly coupling ctf_bufopen and ctf_file_close together. So split the header out into a new ctf_file_t.ctf_header, which is written out explicitly: squeeze it out of the CTF buffer whenever we reallocate it, and use ctf_file_t.ctf_buf to skip past the header when we do not need to reallocate (when no upgrading or endian-flipping is required). We now track whether the CTF base can be freed explicitly via a new ctf_dynbase pointer which is non-NULL only when freeing is possible. With all this done, we can upgrade the header on the fly and add new fields as desired, via a new upgrade_header function in ctf-open. As with other forms of upgrading, libctf upgrades older headers automatically to the latest supported version at open time. For a first use of this field, we add a new string field cth_cuname, and a corresponding setter/getter pair ctf_cuname_set and ctf_cuname: this is used by debuggers to determine whether a CTF section's types relate to a single compilation unit, or to all compilation units in the program. (Types with ambiguous definitions in different CUs have only one of these types placed in the top-level shared .ctf container: the rest are placed in much smaller per-CU containers, which have the shared container as their parent. Since CTF must be useful in the absence of DWARF, we store the names of the relevant CUs ourselves, so the debugger can look them up.) v5: fix tabdamage. include/ * ctf-api.h (ctf_cuname): New function. (ctf_cuname_set): Likewise. * ctf.h: Improve comment around upgrading, no longer implying that v2 is the target of upgrades (it is v3 now). (ctf_header_v2_t): New, old-format header for backward compatibility. (ctf_header_t): Add cth_cuname: this is the first of several header changes in format v3. libctf/ * ctf-impl.h (ctf_file_t): New fields ctf_header, ctf_dynbase, ctf_cuname, ctf_dyncuname: ctf_base and ctf_buf are no longer const. * ctf-open.c (ctf_set_base): Preserve the gap between ctf_buf and ctf_base: do not assume that it is always sizeof (ctf_header_t). Print out ctf_cuname: only print out ctf_parname if set. (ctf_free_base): Removed, ctf_base is no longer freed: free ctf_dynbase instead. (ctf_set_version): Fix spacing. (upgrade_header): New, in-place header upgrading. (upgrade_types): Rename to... (upgrade_types_v1): ... this. Free ctf_dynbase, not ctf_base. No longer track old and new headers separately. No longer allow for header sizes explicitly: squeeze the headers out on upgrade (they are preserved in fp->ctf_header). Set ctf_dynbase, ctf_base and ctf_buf explicitly. Use ctf_free, not ctf_free_base. (upgrade_types): New, also handle ctf_parmax updating. (flip_header): Flip ctf_cuname. (flip_types): Flip BUF explicitly rather than deriving BUF from BASE. (ctf_bufopen): Store the header in fp->ctf_header. Correct minimum required alignment of objtoff and funcoff. No longer store it in the ctf_buf unless that buf is derived unmodified from the input. Set ctf_dynbase where ctf_base is dynamically allocated. Drop locals that duplicate fields in ctf_file: move allocation of ctf_file further up instead. Call upgrade_header as needed. Move version-specific ctf_parmax initialization into upgrade_types. More concise error handling. (ctf_file_close): No longer test for null pointers before freeing. Free ctf_dyncuname, ctf_dynbase, and ctf_header. Do not call ctf_free_base. (ctf_cuname): New. (ctf_cuname_set): New. * ctf-create.c (ctf_update): Populate ctf_cuname. (ctf_gzwrite): Write out the header explicitly. Remove obsolescent comment. (ctf_write): Likewise. (ctf_compress_write): Get the header from ctf_header, not ctf_base. Fix the compression length: fp->ctf_size never counted the CTF header. Simplify the compress call accordingly.	7 years ago
Nick Alcock	f5e9c9bde0	libctf: deduplicate and sort the string table ctf.h states: > [...] the CTF string table does not contain any duplicated strings. Unfortunately this is entirely untrue: libctf has before now made no attempt whatsoever to deduplicate the string table. It computes the string table's length on the fly as it adds new strings to the dynamic CTF file, and ctf_update() just writes each string to the table and notes the current write position as it traverses the dynamic CTF file's data structures and builds the final CTF buffer. There is no global view of the strings and no deduplication. Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding a new dynhash table ctf_str_atoms that maps unique strings to a list of references to those strings: a reference is a simple uint32_t * to some value somewhere in the under-construction CTF buffer that needs updating to note the string offset when the strtab is laid out. Adding a string is now a simple matter of calling ctf_str_add_ref(), which adds a new atom to the atoms table, if one doesn't already exist, and adding the location of the reference to this atom to the refs list attached to the atom: this works reliably as long as one takes care to only call ctf_str_add_ref() once the final location of the offset is known (so you can't call it on a temporary structure and then memcpy() that structure into place in the CTF buffer, because the ref will still point to the old location: ctf_update() changes accordingly). Generating the CTF string table is a matter of calling ctf_str_write_strtab(), which counts the length and number of elements in the atoms table using the ctf_dynhash_iter() function we just added, populating an array of pointers into the atoms table and sorting it into order (to help compressors), then traversing this table and emitting it, updating the refs to each atom as we go. The only complexity here is arranging to keep the null string at offset zero, since a lot of code in libctf depends on being able to leave strtab references at 0 to indicate 'no name'. Once the table is constructed and the refs updated, we know how long it is, so we can realloc() the partial CTF buffer we allocated earlier and can copy the table on to the end of it (and purge the refs because they're not needed any more and have been invalidated by the realloc() call in any case). The net effect of all this is a reduction in uncompressed strtab sizes of about 30% (perhaps a quarter to a half of all strings across the Linux kernel are eliminated as duplicates). Of course, duplicated strings are highly redundant, so the space saving after compression is only about 20%: when the other non-strtab sections are factored in, CTF sizes shrink by about 10%. No change in externally-visible API or file format (other than the reduction in pointless redundancy). libctf/ * ctf-impl.h: (struct ctf_strs_writable): New, non-const version of struct ctf_strs. (struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated. (struct ctf_str_atom): New, disambiguated single string. (struct ctf_str_atom_ref): New, points to some other location that references this string's offset. (struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs. Remove member ctf_dtvstrlen: we no longer track the total strlen as we add strings. (ctf_str_create_atoms): Declare new function in ctf-string.c. (ctf_str_free_atoms): Likewise. (ctf_str_add): Likewise. (ctf_str_add_ref): Likewise. (ctf_str_purge_refs): Likewise. (ctf_str_write_strtab): Likewise. (ctf_realloc): Declare new function in ctf-util.c. * ctf-open.c (ctf_bufopen): Create the atoms table. (ctf_file_close): Destroy it. * ctf-create.c (ctf_update): Copy-and-free it on update. No longer special-case the position of the parname string. Construct the strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the rest of each buffer element is constructed, not via open-coding: realloc the CTF buffer and append the strtab to it. No longer maintain ctf_dtvstrlen. Sort the variable entry table later, after strtab construction. (ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members. (ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref after buffer element construction instead. (ctf_copy_lmembers): Likewise. (ctf_copy_emembers): Likewise. (ctf_create): No longer maintain the ctf_dtvstrlen. (ctf_dtd_delete): Likewise. (ctf_dvd_delete): Likewise. (ctf_add_generic): Likewise. (ctf_add_enumerator): Likewise. (ctf_add_member_offset): Likewise. (ctf_add_variable): Likewise. (membadd): Likewise. * ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts if there are active ctf_str_num_refs. (ctf_strraw): Move to ctf-string.c. (ctf_strptr): Likewise. * ctf-string.c: New file, strtab manipulation. * Makefile.am (libctf_a_SOURCES): Add it. * Makefile.in: Regenerate.	7 years ago
Nick Alcock	7cee18263c	libctf: endianness fixes Testing of the first code to generate CTF_K_SLICEs on big-endian revealed a bunch of new problems in this area. Most importantly, the trick we did earlier to avoid wasting two bytes on padding in the ctf_slice_t is best avoided: because it leads to the whole file after that point no longer being naturally aligned, all multibyte accesses from then on must use memmove() to avoid unaligned access on platforms where that is fatal. In future, this is planned, but for now we are still doing direct access in many places, so we must revert to making ctf_slice_t properly aligned for storage in an array. Rather than wasting bytes on padding, we boost the size of cts_offset and cts_bits. This is still a waste of space (we cannot have offsets or bits in bitfields > 256) but it cannot be avoided for now, and slices are not so common that this will be a serious problem. A possibly-worse endianness problem fixed at the same time involves a codepath used only for foreign-endian, uncompressed CTF files, where we were not copying the actual CTF data into the buffer, leading to libctf reading only zeroes (or, possibly, uninitialized garbage). Finally, when we read in a CTF file, we copy the header and work from the copy. We were flipping the endianness of the header copy, and of the body of the file buffer, but not of the header in the file buffer itself: so if we write the file back out again we end up with an unreadable frankenfile with header and body of different endiannesses. Fix by flipping both copies of the header. include/ * ctf.h (ctf_slice_t): Make cts_offset and cts_bits unsigned short, so following structures are properly aligned. libctf/ * ctf-open.c (get_vbytes_common): Return the new slice size. (ctf_bufopen): Flip the endianness of the CTF-section header copy. Remember to copy in the CTF data when opening an uncompressed foreign-endian CTF file. Prune useless variable manipulation.	7 years ago

1 2

57 Commits (master)