RFC 1868: portability-lint

lang | libs (cfg)

Summary

There has long been a desire to expand the number of platform- and architecture-specific APIs in the standard library, and to offer subsets of the standard library for working in constrained environments. At the same time, we want to retain the property that Rust code is portable by default.

This RFC proposes a new portability lint, which threads the needle between these two desires. The lint piggybacks on the existing cfg system, so that using APIs involving cfg will generate a warning unless there is explicit acknowledgment of the portability implications.

The lint is intended to make the existing std::os module obsolete, to allow expansion (and subsetting) of the standard library, and to provide deeper checking for portability across the ecosystem.

Motivation

Background: portability and the standard library

One of the goals of the standard library is to provide an interface to hardware and system services. In doing so, there were several competing principles that we wanted to embrace:

The way we balanced these principles was roughly as follows:

Problems with the status quo

The above strategy has served us fairly well in the first year since Rust 1.0, but it's increasingly holding us back from enhancements we'd like to make. It's also suboptimal in a few ways, even for the needs it covers.

Problems with std::os:

Problems with libcore/the facade:

What are our portability goals?

Taking a step back from the specific problems with the status quo, it's worth thinking about what it means for Rust to be "portable", and what is realistic to achieve. We should be asking this question not just for the standard library, but for the Rust library ecosystem in general.

The premise of this RFC is that there are roughly three desired portability levels for a library. In order of increasing portability:

There's a fundamental tradeoff here. On the one hand, we want Rust libraries to be as portable as possible. On the other hand, achieving maximal portability can be a big burden for library authors. Our approach so far has been to identify "mainstream platform assumptions", as mentioned above, and guide code to work on all mainstream platforms by default; by convention, such portability is the default expectation of libraries on crates.io. This RFC formalizes that approach in a deeper way.

An important point: while we can expect library authors who are striving for portability to test their code on a variety of target platforms, we can't make that assumption for the average library. In other words, if we want to guide all Rust code toward at least mainstream portability, we will need to do so in a way that doesn't require actually compiling and testing for all mainstream scenarios.

Detailed design

The basic idea

The core problem we want to solve is:

The core idea is that having to write cfg is a sufficient speedbump, as it makes explicit what platform assumptions a piece of code is making. But today, you don't have to be within a cfg to call something labeled with cfg.

Let's take a concrete example: the as_raw_fd method. We'd like to provide this API as an inherent method on things like files. But it's not a "mainstream" API; it only works on Unix. If you tried to use it and compiled your code on Windows, you would discover the problem right away, since the API would not be available due to cfg. But if you were only testing on Linux, you might never notice, since the API is available there.

The basic idea of this RFC is to provide an additional layer of checking on top of the existing cfg system, to avoid usage of an API accidentally working because you happen to be compiling for a given target platform. This checking is performed through a new portability lint, which warns when invoking APIs marked with cfg unless you've explicitly acknowledged the portability implications. We'll see how you do that in a moment.

Going back to our example, we'd like to define methods on File like:

impl File {
    #[cfg(unix)]
    fn as_raw_fd(&self) -> RawFd { ... }

    #[cfg(windows)]
    fn as_raw_handle(&self) -> RawHandle { ... }
}

If you attempted to call as_raw_fd, when compiling on Unix you'd get a warning from the portability lint that you're calling an API not available on all mainstream platforms. There are basically three ways to react (all of which will make the warning go away):

In code, we'd have:

////////////////////////////////////////////////////////////////////////////////
// The code we might have written initially:
////////////////////////////////////////////////////////////////////////////////

fn unlabeled() {
    // Would generate a warning: calling a `unix`-only API while only
    // assuming a mainstream platform
    let fd = File::open("foo.txt").unwrap().as_raw_fd();
}

////////////////////////////////////////////////////////////////////////////////
// Code that opts into platform-specificness:
////////////////////////////////////////////////////////////////////////////////

#[cfg(unix)]
fn foo() {
    // No warning: we're within code that assumes `unix`
    let fd = File::open("foo.txt").unwrap().as_raw_fd();
}

#[cfg(windows)]
fn foo() {
    // No warning: we're within code that assumes `windows`
    let handle = File::open("foo.txt").unwrap().as_raw_handle();
}

#[cfg(linux)]
fn linux_only() {
    // No warning: we're within code that assumes `linux`, which implies `unix`
    let fd = File::open("foo.txt").unwrap().as_raw_fd();
}

////////////////////////////////////////////////////////////////////////////////
// Code that provides a cross-platform abstraction
////////////////////////////////////////////////////////////////////////////////

// No `cfg` label here; it's a cross-platform function, which we claim
// via the `allow`
#[allow(nonportable)]
fn cross_platform() {
    // invoke an item with a more restrictive `cfg`
    foo()
}

As with many lints, the portability lint is best effort: it is not required to provide airtight guarantees about portability. However, the RFC sketches a plausible implementation route that should cover the vast majority of cases.

Note that this lint will only check code that is actually compiled on the current platform, so the following code would not produce a warning when compiled on unix:

pub fn mycrate_function() {
    // ...
}

#[cfg(windows)]
pub fn windows_specific_mycrate_function() {
    // this call should warn since it makes an additional assumption
    windows_more_specific_mycrate_function();
}

#[cfg(all(windows, target_pointer_width = "64"))]
pub fn windows_more_specific_mycrate_function() {
    // ...
}

However, any such "missed portability issues" are only possible when already using cfg, which means a "speedbump" has already been passed.

With that overview in mind, let's dig into the details.

The lint definition

The lint is structured somewhat akin to a type and effect system: roughly speaking, items that are labeled with a given cfg assumption can only be used within code making that same cfg assumption.

More precisely, each item has a portability, consisting of all the lexically-nested uses of cfg. If there are multiple uses of cfg, the portability is taken to be their conjunction:

#[cfg(unix)]
mod foo {
    #[cfg(target_pointer_width = "32")]
    fn bar() {
        // the portability of `bar` is `all(unix, target_pointer_width = "32")`
    }
}

The portability only considers built-in cfg attributes (like target_os), not Cargo features (which are treated as automatically true for the lint purposes).

The lint is then straightforward to define at a high level: it walks over item definitions and checks that the item's portability is narrower than the portability of items it references or invokes. For example, bar in the above could invoke an item with portability unix and/or target_pointer_width = "32", but not one with portability linux.

To fully define the lint, though, we need to give more details about what "narrower" means, and how referenced item portability is determined.

Comparing portabilities

What does it mean for a portability to be narrower? In general, portability is a logical expression, using the operators all, any, not on top of primitive expressions like unix. Portability P is narrower than portability Q if P implies Q as a logic formula.

In general, comparing two portabilities is equivalent to solving SAT, an NP-complete problem -- a frightening prospect for a lint! However, note that worst-case execution is exponential in the number of variables (i.e., primitive cfg constraints), not the number/complexity of clauses, and most comparisons should involve a very small number of variables. We can likely get away with a naive SAT implementation, perhaps with a handful of optimiziations specific to our use-case. In the limit, there are also many well-known techniques for solving SAT efficiently even on very large examples that arise in real-world usage.

Axioms

Another aspect of portability comparison is the relationship between things like unix and linux. In logical terms, we want to assume that linux implies unix, for example.

The primitive portabilities we'll be comparing are all built in (since we are not including Cargo features). The solver can thus build in a number of assumptions about these portabilities. The end result is that code like the following should pass the lint:

#[cfg(unix)]
fn unix_only() { .. }

#[cfg(linux)]
fn linux_only() {
    // permitted since `linux` implies `unix`
    unix_only()
}

Of course, primitive portabilities in practice are key-value pairs (like target_os = "unix"). This RFC proposes to treat all keys as multimaps, that is, to not introduce assumptions like nand(target_os = "unix", target_os = "windows") for simplicity's sake; uses of cfg in practice will not produce such nonsensical situations. However, the precise details of how these implications are specified---and what implications are desired---are left as implementation details that need to be worked out with real-world experience.

Determining the portability of referenced items

How is the portability of a referenced item determined? The lint will resolve an item to its definition, and use the portability of that definition, which will be recorded in metadata. For the case of trait items, however, this will involve attempting to resolve the invocation to a particular impl, to look up the portability of that impl. We can set up trait selection to yield portability information with the selected impl, which will allow us to catch cases like the following:

trait Foo {
    fn foo();
}

struct MyType;

#[cfg(unix)]
impl Foo for MyType {
    fn foo() { .. }
}

fn use_foo<T: Foo>() {
    T::foo()
}

fn invoke() {
    // invokes a `cfg(unix)` item via a generic function, but we can catch it
    // when checking that `MyType: Foo`, since selection will say that we need
    // our context to imply `unix`
    use_foo::<MyType>();
}

The story for std

With these basic mechanisms in hand, let's sketch out how we might apply them to the standard library to achieve our initial goals. This part of the RFC should not be considered normative; it's left to the implementation to make the final determination about how to set up the standard library.

The mainstream platform

The "mainstream platform" will be expressed via a new primitive cfg pattern called std. This is the default portability of all crates, unless opted-out (see below on "subsetting std"). Likewise, most items in std will initially be exported at std portability level (but see subsets below). These two facts together mean that existing uses of std will continue to work without issuing any warnings.

Expanding std

With the above setup, handling extensions to std with APIs like as_raw_fd is straightforward. In particular, we can write:

impl File {
    #[cfg(unix)]
    fn as_raw_fd(&self) -> RawFd { ... }

    #[cfg(windows)]
    fn as_raw_handle(&self) -> RawHandle { ... }
}

and the portability of as_raw_fd will be all(std, unix). Thus, any code using as_raw_fd will need to be in a unix context in particular.

We can thus deprecate the std::os module in favor of these in-place APIs. Doing so leverages the fact that we're using a portability lint: these new inherent methods will shadow the existing ones in std::os, and may generate new warnings, but this is considered an acceptable change. After all, lints on dependencies are automatically capped, and the lint will not prevent code from compiling--and can be silenced.

For hardware features like additional atomics or SIMD, we can use the target_feature cfg key to label the APIs -- which has to be done anyway, but will also do the right thing for the lint.

In short, for expansions there's basically nothing to do. You just add the API in its natural location, with its natural cfg, and everything works out.

Subsetting std

What about subsets of std?

What use case do we want to address? Going back to the Portability Goals discussed earlier, the goal of subsetting std is mostly about helping people who want maximum portability. For this use case, you should opt out of the mainstream platform, and then whitelist the various features you need, thus giving you assistance in using the minimal set of assumptions needed.

Opting out of the mainstream platform. To opt out of the std platform, you can just apply a cfg to your crate definition. The assumptions of that cfg will form the baseline for the crate.

Carving up std into whitelistable features. When we want to provide subsets of std, we can introduce a new set of target features, along the following lines:

To introduce these features, we would change APIs in std from being marked as #[cfg(std)] to instead being labeled with the particular feature, e.g.:

// previously: #[cfg(std)]
#[cfg(target_feature = "thread")]
mod thread;

// previously: #[cfg(std)]
#[cfg(target_feature = "fs")]
mod fs;

and so on. We can then set up axioms such that std implies all of these features. That way existing code written at the default portability level will not produce warnings when using the standard library. And in general, we can carve out increasingly fine-grained subsets, setting up implications between the previous coarse-grained features and the new subsets.

On the other side, library authors shooting for maximal portability should opt out of cfg(std), and use cfg as little as possible, adding features to their whitelist only after deciding they're truly needed, or abstracting over them (such as using threading for parallelism only when it was available).

Proposed rollout

The most pressing problem in std is the desire for expansion, rather than subsetting, so we should start there. The cfg needed for expansion is totally straightforward, and will allow us to gain experience with the lint.

Later, we can start exploring subsets of std, which will likely require some more thoughtful design to find the right granularity.

Drawbacks

There are several potential drawbacks to the approach of this RFC:

The fact that it's a lint offers some help with the first two points; the use of std as a default portability level should also help quite a bit with pedanticness.

The worry about SAT solving is harder to mitigate; there's not much concrete evidence in either direction. But it is yet another place where the fact that it's a lint could help: we may be able to simply skip checking pathological cases, if they indeed arise in practice. In any case, it's hard to know how concerned to be until we try it.

While the fact that it's a lint gives us more leeway to experiment, it's also a lint that could produce widespread warnings throughout the ecosystem, so we need to exercise care.

Alternatives

The main alternatives are:

How we teach this

For people simply using libraries, this feature "teaches itself" by generating warnings. Those warnings should make clear what to do to fix the problem, and ideally provide extended error information that describes the system in more detail.

For library authors, the documentation for cfg and match_cfg would explain the implications for the lint, and walk through several examples illustrating the scenarios that arise in practice.

Unresolved questions

Extensions to cfg itself

If we allow cfg to go beyond simple key-value pairs, for example to talk about ranges, we will need to accommodate that somehow in the lint. One plausible approach would be to use something more like SMT solving, which incorporates reasoning about things like ordering constraints in addition to basic SAT questions.

External libraries

It's not clear what the story should be for a library like libc, which currently involves intricate uses of cfg. We should have some idea for how to approach such cases before landing the RFC.

The standard library

To what extent does this proposal obviate the need for the std facade? Might it be possible to deprecate libcore in favor of the "subsetting std" approach?

Cargo features

It's unclear whether, or how, to extend this approach to deal with Cargo features. In particular, features are namespaced per crate, so there's no way to use the cfg system today to talk about upstream features.

Appendix: possible extensions

match_cfg

The original version of this RFC was more expansive, and proposed a match_cfg macro that provided some additional checking.

The match_cfg macro takes a sequence of cfg patterns, followed by => and an expression. Its syntax and semantics resembles that of match. However, there are some special considerations when checking portability:

Thus, for example, the following code will pass the lint:

#[cfg(windows)]
fn windows_only() { .. }

#[cfg(unix)]
fn unix_only() { .. }

#[cfg(any(windows, unix))]
fn portable() {
    // the expression here has portability `any(windows, unix)`
    match_cfg! {
        windows => {
            // allowed because we are within a scope with
            // portability `all(any(windows, unix), windows)`
            windows_only()
        }
        unix => {
            // allowed because we are within a scope with
            // portability `all(any(windows, unix), unix)`
            unix_only()
        }
    }
}

If you have a match_case that covers all cases (like windows and not(windows)), then it imposes no portability constraints on its context.

On more reflection, though, this extension doesn't seem so worthwhile: while it provides some additional checking, the fact remains that only the currently-enabled cfg is fully checked, so the additional guarantee you get is somewhat mixed. It's also a rare (maybe non-existent) error to explicitly write code that's broken down by platforms, but forget one of the platforms you wish to cover.

We can, however, add match_cfg as a backwards-compatible extension at any time.

Copyright 2022 Nick Cameron and the RFC authors.