Failure due to system libs now being older

I provide a Rust binding for PulseAudio. Lately all of my builds are failing due to PulseAudio being too old a version.

As a binding it makes use of course of the entire system lib’s C API. It has feature flags to control which version of PA it is compiled for, either 8-11 or 12+, where 12+ uses the extra (just 1) symbol v12 introduced.

All of my builds are now failing telling me that this symbol does not exist in the installed system lib, thus that it is older than v12.

Xenial supposedly was released with PA v8, and Bionic with PA v11.

Previously the builds all worked fine. I’ve had to introduced odd fixes for mistakes I’ve made in the travis config and per changes to feature flags in the project, but looking over it I don’t see anything that would suggest that builds were not legitimately being used with PA v12 on travis in the past. So what changed?

I know I can fix the failures by limiting testing to PA <=11 only, either with xenial or bionic (no relevant changes in PA from 8 though 11), but I had PA 12 testing also before, and I’d like it back ideally; I just don’t know what happened…?

Comparing your last successful build and the first failed one, I see lots of changes. Any of them can be the cause.

  • You last successful build has run in Trusty, so this is (another) case of
  • As such, Rust version and most other versions have changed between them.

  • Then, there are lots of source code changes. You are even building stuff with different flags

  • Finally, AFAICS your build.rs files don’t bail out if libpulse is too old. Instead, they supply some default value (a non-versioned lib reference or something, I presume). Trusty presumably has a version older than 8.0, so I suppose that clause got used.

Hmm. Fyi, your comparison is crossing branches (master to next).

I can see that indeed the last success was on a Trusty environment, while the failures begin on Xenial. Note that there was a break of a few months between runs during which the switch to Xenial occurred.

Trusty I see actually was released with PulseAudio v4.x, and that’s the version that was being installed by APT according the the log (according to the package version). New function symbols were introduced in PA since v4.x which are made use of by my library, and missing symbols in the installed system library result in link failure…

That confuses me as to why on earth things worked at all on the Trusty setup… It would have needed to have PA v8.0+ for any cargo test --all command (which involves linking unlike build) to be successful at all…

Regarding my build.rs falling back to a default value: When I started the project I did not have the PA dev package installed, thus no *.pc files for the PA system libs, thus relying upon pkg-config was a no go. When pkg-config support was added, that was left as a fallback, for those similarly lacking the dev package.

Sidenote: perhaps adding installation of the AP dev package into the travis environment to get the *.pc files would be a good move?

Regarding changes and building with different flags:

  1. as pointed out above, you’re comparing master with next, and next indeed has significant changes to the flags, but there are far fewer such changes on master, which also is failing.

  2. up to the point in the commit history of the last success i was making a mistake with my use of Cargo workspaces wrt. feature flags. The Cargo workspace was very minimal, just referencing the enclosed crates), and as you could see in the travis script I was using features with the expectation that they would just be passed on to them.

I fixed this (on master) by changing the workspace to a full lib-crate model, which depended on the enclosed crates and mapped the feature flags to them. This change was pushed recently, after the Xenial upgrade and is included in the failures.

if we review how things worked though at the point of the last success, you can see in the log that the pa_v12_compatibility flag is getting passed through, enabling PA v12 features, which worked on the old Trusty setup. Regardless, we come back to the fact that we were linking successfully code that uses PA v8+ symbols (the minimum supported version) on an environment apparently with only PA v4.x, which makes no sense…

So did the environment really have PA v4.x, or did it actually somehow have PA v12? If the former, then I don’t understand at all why cargo test commands (which involved linking) were successful, compared to now where the builds are failing complaining about the missing symbols.

Going back to build.rs and use of pkg-config, it surely cannot be the case that in the old setup this stuff was “bailing out” cleanly in seeing too old a version of PA. There is no dev package installed in the environment, so the pkg-config check would fail, going back to the basic fallback. It should thus find the lib, but fail to link due to missing symbols, just as seen in the failures.

I searched “pa_v12_compatibility” in your codebase to assess its effect, and it appeared to me that due to the default clause, it didn’t actually have any effect in Trusty. There are also suspiciously many skipped tests.
So it’s possible that missing symbols in the library older than 8.0 don’t actually result in link errors and/or test failures, the new features are just silently disabled or not used.

Anyway, the question was “what changed” and I believe I answered this. You can fall back to Trusty with dist: trusty to diagnose why it worked there and stuff; you can also request debug mode for your repo.
I’m not familiar with Rust so I really can’t help you much further here.

Installing dev packages is not just a “good move”, they are generally considered a requirement to build native code against the library! You are certainly overthinking it if you are trying to make your build work without them.

If I understand correctly, you’re suggesting that for each test binary that gets built and run by the test suite, it strips out the portion of my bindings library that is not in actual use by that test binary, which happens before linking to the system lib, thus no linking error. That makes a lot of sense.

I’m not certain why then i get the failure regarding the missing symbol from the PA v12 features, since there’s no specific test relating to those features, but maybe its a quirk/bug of the way test binaries and features interact, that it is somehow retaining the method that calls that symbol?

edit: just to clarify that, the change to use of feature flags explains why this problem occurs now and not previously; it’s understanding why a missing symbol occurs in relation to use of this specific PA-12-introduced symbol (only used behind a feature flag) results in a missing symbol error and thus is not being stripped out before linking within tests, when the other symbols are being stripped out. If i remember correctly, there’s no test using that symbol, but perhaps I’m being forgetful, I need to review…

I’ve got some test branches to experiment a little with going back to trusty and such, I’ll push them and see what happens shortly.

Regarding dev packages, as I see it, the primary purpose of them is distribution of C/C++ header files, required when building code that use their respective codebases, at least in C/C++ projects. *.pc files happen to be bundled in with them. In the Rust world there is no need at all for those header files, since the interface must be described in Rust code (done in *-sys crates). The only benefit then to dev packages is the *.pc file for assisting in locating the system lib to link against. When approaching a new Rust project that talks to PulseAudio for instance, you’d simply add a dependency upon libpulse-binding in your project’s toml file, and Cargo downloads and compiles it in along with any dependencies it has and so on. It just feels entirely unnatural to have to then go and separately install a set of dev packages that correspond to the *-sys packages you are making use of, just so that system libraries can be successfully found at the linking stage. My feeling is that in this new world of Rust, distros should move packaging the *.pc files now to the non-dev packages. they’re tiny enough to not make a difference on non-dev systems, and fix this issue with Rust development.

I’m going to go now and play with a few different tweaked branches to see what results I get to see what the impact is, I suspect though that your explanation regarding binaries stripping stuff out is the crux of the matter, so thanks for suggesting that :slight_smile:

small update: so the simple direct fix to get things passing again was obviously to just disable PA v12 features for the tests. I pushed such a change, but it has not worked. testing upon my local machine I see that my previous fix for feature flags to be passed through from the cargo workspace level does not work correctly in cargo test mode… if (on master, not next where flags have changed) I run cargo build --no-default-features --verbose then there is correctly no sign of the PA v12 feature flag and so it is being excluded. If however I run cargo test --no-default-features --verbose --all then I clearly see the PA v12 feature being enabled in tests…

update to that: it seems that using cargo’s --all flag does not work as expected with the new lib-style workspace setup, it adds the default flags of the member crates, overriding the
fact that you’ve disabled them in the dependencies… so removing the use of --all now correctly allows disabling the PA v12 stuff, getting rid of the missing symbol issue relating to it.

now i’ve just got to experimentally determine the min supported version of a new glib dependency to fix the last thing stopping success…

so…

  1. things are not fixed after all, it seems that dropping --all from cargo test with the [package] based workspace toml just results in tests being run on the workspace lib itself, not the members, so is a false result. i have not been able to determine how to correctly marry cargo workspaces+feature control. i will have to research and possibly post on cargo’s issue tracker.

  2. back to the missing symbol issue, having done some more playing… check out the following set of builds:
    Trusty (PA v4): https://travis-ci.org/jnqnfe/pulse-binding-rust/jobs/572936139
    Xenial (PA v8): https://travis-ci.org/jnqnfe/pulse-binding-rust/jobs/572946063
    Bionic (PA v11): https://travis-ci.org/jnqnfe/pulse-binding-rust/jobs/572936129

The only difference here is the travis environment selected. All three are using the same versions of rustc and cargo. Trusty passes, whilst the other two fail, complaining about the missing symbol

ok,

  1. research tells me that actually workspace+feature control is not actually supported yet :confused

  2. i’ve reported the missing symbol issue here: https://github.com/rust-lang/rust/issues/63645