Next Generation Erlang Compiler

[ferd]

Discussion brought in by @jhogberg:

Current model is that the Erlang compiler only looks at the file being compiled, leaving “linking” for later (load time), which is somewhat similar to C and works well in makefiles and manual command-line builds. This is fine in principle.

Unfortunately -include(), -include_lib(), parse transformations, and -behavior() look beyond the file in ill-defined ways which complicates dependency management and makes it difficult to parallelize builds. We can’t tear these features out altogether, but they could be better supported with a new build model.

We hope to start exploring this soon after OTP 24 is released, and I’d like to have the WG on board from the start.

Following the WG meeting, we felt that we should bring in the discussion about requirements and prior art across languages in this thread.

People of interest at first: @jhogberg, @josevalim @Ostera, @ferd, @tsloughter, anyone interested.

[jhogberg]

Unfortunately -include(), -include_lib(), parse transformations, and -behavior() look beyond the file in ill-defined ways which complicates dependency management and makes it difficult to parallelize builds. We can’t tear these features out altogether, but they could be better supported with a new build model.

It's worth noting that these issues arise before the compiler runs: -include() / -include_lib() are handled by the pre-processor, parse transformations are invoked after that, and finally -behaviour() is handled by erl_lint.

One solution would be to move the responsibility for these over to the build tools (whether rebar3 or plain make), having them invoke the pre-processor and parse transformations themselves before passing the files/forms over to the compiler. rebar3 does much of this already when building its dependency graph so I don't think it would require much work on either side.

(This would of course be optional, plain erlc has to work the same way as before)

Following the WG meeting, we felt that we should bring in the discussion about requirements and prior art across languages in this thread.

One thing I'd like to look into is adding an official project model, similar to Elixir. This lets them perform cross-module optimizations ("protocol consolidation"?) and various checks at compile time, which otherwise require separate runs of external tools like xref or dialyzer.

Having a similar model would let us share much of that work and opens the door to more aggressive optimizations (without necessarily sacrificing hot code upgrades 😎), but it's a lot more work than just fixing the above problems. I'd love to hear any ideas you have in this area.

[tsloughter]

I was coming to say the same thing, basically make pare transform like rebar3 plugins, meaning they must be built prior to building of the application.

This would make for less parallelization but assuming transforms are small and rare it may be worth moving them out like that.

[jhogberg]

I like the idea of defining which transformations an application needs, but executing compiled ones means we still need to deal with the code path and have fun loading/unloading modules to handle version clashes. It's not impossible to deal with but I figured it's worth mentioning.

[ferd]

parse-transform as a dep would have the unhappy side-effect of having to test your parse transform in a different application that the one that defines it, which under the current publishing model we have for deps (deps can't be many apps) means your tests can't easily live with your source code force parse transforms... This might be an acceptable tradeoff, but I wanted to point out the pain point it might create.

[michalmuskala]

Wouldn't it work if we treated tests as a separate "application" to the main code, effectively making the main code a dependency of the tests

[ferd]

It would, but I mean quite literal applications with a .app file. Currently tests are just "extra" code that belongs to an app but don't get bundled in its .app file. We could arguable create virtual test apps, but this would mean, for example, that direct -include(...) directives ought to not be able to see the private includes of the app being tested at the time unless we break these rules for special test apps in the compiler's scoping...

[max-au]

For example, if you had to explicitly list all parse transformations and behaviors you'd like to use in the project definition

Do you mean static mapping like this: #{gen_server => "stdlib/gen_server.erl"} in project configuration? That's in fact what rebar3 does (dynamically builds this map), but it needs to know version and filesystem location of "stdlib", which means scanning directory structure.

This may introduce mixing of layers - "application" and "project". As a developer I want to use some behaviour/transformation in my app, and ship it this way. It may be hard to re-use an application in multiple projects if behaviour => source mapping is not defined in the application itself. I can think of adding this information to app.src, to keep application self-contained.

However in order to properly support this, app.src would need to receive some concept of "profiles". For example, in "test" profile I want to add some parse_transform (from eunit, or pre-compile all files with cover). Same for dependencies, I do want to have "meck" in test, but not in production.

[jhogberg]

Do you mean static mapping like this: #{gen_server => "stdlib/gen_server.erl"} in project configuration? That's in fact what rebar3 does (dynamically builds this map)

More or less.

but it needs to know version and filesystem location of "stdlib", which means scanning directory structure.

Not if the version is specified at the project ("release") level. An alternative would be for the build tool to figure out dependency versions once per build. I'm not going to tie myself down to any particular solution here but I'd like to see versions defined up front so that builds have a single known version of each (runtime) dependency, if that makes any sense.

This may introduce mixing of layers - "application" and "project". As a developer I want to use some behaviour/transformation in my app, and ship it this way. It may be hard to re-use an application in multiple projects if behaviour => source mapping is not defined in the application itself. I can think of adding this information to app.src, to keep application self-contained.

We already mix layers: the version of each dependency is largely taken from the code path. I propose taking the version of each (runtime) dependency from a project/build definition instead, and it wouldn't prevent defining behavior => source mappings in app.src (#{gen_server => "stdlib/gen_server.erl"} could be relative to the stdlib version of the current build).

[jhogberg]

Sorry about the late reply, rc1 has been keeping me busy. :/

Currently it's done by scanning filesystem folders and reading app.src files. Large projects may contain 1000+ applications, so application_discovery step becomes a bottleneck. The missing bit is mapping between application name, and actual location on disk.

Looking at large-scale build tools, they all have strict naming conventions between target names (applications) and filesystem location. Often application name is namespaced. I can imagine something like -include_lib("otp/kernel/file.hrl"). or -behaviour("myproject/mysubproj/myapp/gen_my.erl"). This would allow to avoid file tree scanning.

I don't see how this helps avoid walking the filesystem. Aren't build tools like rebar3 already aware of all applications (rebar.config or similar) and where they are on disk (copied to the build folder)? Is this insufficient somehow? I feel like I'm missing something.

[max-au]

Aren't build tools like rebar3 already aware of all applications (rebar.config or similar) and where they are on disk (copied to the build folder)?

The very first step (application_discovery) is disk scan: https://github.com/erlang/rebar3/blob/master/src/rebar_app_discover.erl#L258

Currently applications do not have structured namespaces, or "packages" as in Java. So mapping between application name and location on disk has to be discovered by scanning folders.

[jhogberg]

I can see how stricter structure would reduce the amount of scanning we have to do, but I don't see how it would avoid it altogether. Even if we say that "all applications are folders in the project root and their appname.src.in file lives in appname/src" we'd still need to scan the project root to figure out what applications we have.

[jhogberg]

Mind you, I'm not against structuring things that way, I'm just curious as to the benefits and want to make sure I grasp it all. Reducing scanning is good enough on its own but if it's possible to get rid of it altogether I'd like to know what I'm missing.

[max-au]

Just looking at other build systems, they don't scan project root to find all applications.
Instead, they start with some target. For example, a release (or target system). That release has dependencies on several applications. Build tool does not need to scan directories to find $app.app files, if paths are known.

However requiring all applications to be in a single directory is not desirable. We have short of 400 apps, structured in a dozen sub-folders (some approximation of Java packages again).

[leostera]

Hola! 👋🏽 I've been meaning to reply but haven't managed to until now 🙈. I saw Maxim's answer and it touches the core issues I've had with the BEAM build tooling thus far, so I'll try to explain. Yes, build systems like Bazel, Buck, or Pants will not do file scanning because that just does not scale. It does not scale in a single machine, and it certainly does not scale for distributed builds. Instead, these tools end up building the dependency graph lazily, from a reverse-topological path (starting at the target), and deferring the current target to a queue whenever they find an unmet dependency that needs to be built first. So instead of scanning, building a dependency tree, and then scoping the path (A -> B -> C) when you ask for C, they just immediately begin at C and ask: what does C need to be built? B. Okay lets build B instead and queue C for later. Repeat until you find leaf targets that have no dependencies. This also builds a queue in topo-sort that can be consumed by N-workers parallelly. This gives you "O(1) builds" that scale linearly with respect to the length of the path of the target being built. Throw in a content-addressable cache for the targets and you get pretty damn close to instant rebuilds at large scales. That's what you want. Sadly these tools have a large upfront adoption cost since they force you as a user to explicitly specify _every dependency_. Parse transforms, include paths, env variables, everything should be explicitly annotated as a dependency to a target. Of course this is work done by a tool outside of the compiler, but it means that the compiler should be able to be pointed to the right things explicitly, and fail if they aren't there. Eg, include paths could be "logical" (relative to the build system's view of the world) to support things like remote build execution or distributed caches without a lot of edge-casing. I've found myself vendoring and rewriting paths in libraries to make them work with a prototype build system for the BEAM that follows the philosophy mentioned above. Again, sorry I couldn't get involved in the discussion earlier, and I hope you find this input useful! 🙌🏽. Side-note: I've been prototyping such a tool in the Zap project ( https://github.com/AbstractMachinesLab/zap) in the hopes of showcasing it in our next meeting but I haven't been able to implement the last 20% that actually gets us the "O(1) builds".

…

On Mon, Mar 1, 2021 at 10:46 PM Maxim Fedorov ***@***.***> wrote: Just looking at other build systems, they don't scan project root to find all applications. Instead, they start with some target. For example, a release (or target system). That release has dependencies on several applications. Build tool does not need to scan directories to find $app.app files, if paths are known. However requiring all applications to be in a single directory is not desirable. We have short of 400 apps, structured in a dozen sub-folders (some approximation of Java packages again). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#16 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGQRTTUDVHQEVRJAAS5GN3TBQDKVANCNFSM4XE7ZHTQ> .

-- / Leandro

[leostera]

I should add that in the OCaml ecosystem the dune (https://dune.build) tool has done marvelous work at establishing new conventions and adopting existing ones to make the benefits of tools like Bazel or Buck very very cheap. Adding the equivalent of a parse transform is a matter of dropping in a line like: (preprocessors (pps ...)) in your library's dune file: # this is the file path/to/lib/dune (library (name my_lib) (libraries my_dep1 my_dep2) (preprocessors (pps my_parse_transform))) I've been building a rather large project with it (Caramel currently has ~5000 build targets, including vendored dependencies) and the experience has been excellent. But the OCaml compiler is rather dumb when it comes to looking for things! So maybe keeping erlc dumber is better in the long haul.

…

On Mon, Mar 1, 2021 at 11:16 PM Leandro Ostera ***@***.***> wrote: Hola! 👋🏽 I've been meaning to reply but haven't managed to until now 🙈. I saw Maxim's answer and it touches the core issues I've had with the BEAM build tooling thus far, so I'll try to explain. Yes, build systems like Bazel, Buck, or Pants will not do file scanning because that just does not scale. It does not scale in a single machine, and it certainly does not scale for distributed builds. Instead, these tools end up building the dependency graph lazily, from a reverse-topological path (starting at the target), and deferring the current target to a queue whenever they find an unmet dependency that needs to be built first. So instead of scanning, building a dependency tree, and then scoping the path (A -> B -> C) when you ask for C, they just immediately begin at C and ask: what does C need to be built? B. Okay lets build B instead and queue C for later. Repeat until you find leaf targets that have no dependencies. This also builds a queue in topo-sort that can be consumed by N-workers parallelly. This gives you "O(1) builds" that scale linearly with respect to the length of the path of the target being built. Throw in a content-addressable cache for the targets and you get pretty damn close to instant rebuilds at large scales. That's what you want. Sadly these tools have a large upfront adoption cost since they force you as a user to explicitly specify _every dependency_. Parse transforms, include paths, env variables, everything should be explicitly annotated as a dependency to a target. Of course this is work done by a tool outside of the compiler, but it means that the compiler should be able to be pointed to the right things explicitly, and fail if they aren't there. Eg, include paths could be "logical" (relative to the build system's view of the world) to support things like remote build execution or distributed caches without a lot of edge-casing. I've found myself vendoring and rewriting paths in libraries to make them work with a prototype build system for the BEAM that follows the philosophy mentioned above. Again, sorry I couldn't get involved in the discussion earlier, and I hope you find this input useful! 🙌🏽. Side-note: I've been prototyping such a tool in the Zap project ( https://github.com/AbstractMachinesLab/zap) in the hopes of showcasing it in our next meeting but I haven't been able to implement the last 20% that actually gets us the "O(1) builds". On Mon, Mar 1, 2021 at 10:46 PM Maxim Fedorov ***@***.***> wrote: > Just looking at other build systems, they don't scan project root to find > all applications. > Instead, they start with some target. For example, a release (or target > system). That release has dependencies on several applications. Build tool > does not need to scan directories to find $app.app files, if paths are > known. > > However requiring all applications to be in a single directory is not > desirable. We have short of 400 apps, structured in a dozen sub-folders > (some approximation of Java packages again). > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#16 (reply in thread)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAGQRTTUDVHQEVRJAAS5GN3TBQDKVANCNFSM4XE7ZHTQ> > . > -- / Leandro

-- / Leandro

[leostera]

Even if we say that "all applications are folders in the project root and their appname.src.in file lives in appname/src" we'd still need to scan the project root to figure out what applications we have.

Calling `rebar3 build //apps/my_app` would know exactly where to start building. You'd read the `app.src` file right at `./apps/my_app/app.src` and let the dependency graph take you where you need to go. If all paths are logical (absolute but in relation to the root of the workspace), then you can just read the exact paths to the dependencies. Both on local or remote builds. This would mean you'd have to specify dependencies with a path instead. So no `cowboy` but `//3rdparty/beam/cowboy`. And yes, this would mean that you can have multiple applications with the same name.

…

On Mon, Mar 1, 2021 at 11:22 PM Leandro Ostera ***@***.***> wrote: I should add that in the OCaml ecosystem the dune (https://dune.build) tool has done marvelous work at establishing new conventions and adopting existing ones to make the benefits of tools like Bazel or Buck very very cheap. Adding the equivalent of a parse transform is a matter of dropping in a line like: (preprocessors (pps ...)) in your library's dune file: # this is the file path/to/lib/dune (library (name my_lib) (libraries my_dep1 my_dep2) (preprocessors (pps my_parse_transform))) I've been building a rather large project with it (Caramel currently has ~5000 build targets, including vendored dependencies) and the experience has been excellent. But the OCaml compiler is rather dumb when it comes to looking for things! So maybe keeping erlc dumber is better in the long haul. On Mon, Mar 1, 2021 at 11:16 PM Leandro Ostera ***@***.***> wrote: > Hola! 👋🏽 > > I've been meaning to reply but haven't managed to until now 🙈. I saw > Maxim's answer and it touches the core issues I've had with the BEAM build > tooling thus far, so I'll try to explain. > > Yes, build systems like Bazel, Buck, or Pants will not do file > scanning because that just does not scale. It does not scale in a single > machine, and it certainly does not scale for distributed builds. > > Instead, these tools end up building the dependency graph lazily, from a > reverse-topological path (starting at the target), and deferring the > current target to a queue whenever they find an unmet dependency that needs > to be built first. > > So instead of scanning, building a dependency tree, and then scoping the > path (A -> B -> C) when you ask for C, they just immediately begin at C and > ask: what does C need to be built? B. Okay lets build B instead and queue C > for later. Repeat until you find leaf targets that have no dependencies. > This also builds a queue in topo-sort that can be consumed by N-workers > parallelly. This gives you "O(1) builds" that scale linearly with respect > to the length of the path of the target being built. > > Throw in a content-addressable cache for the targets and you get pretty > damn close to instant rebuilds at large scales. That's what you want. Sadly > these tools have a large upfront adoption cost since they force you as a > user to explicitly specify _every dependency_. Parse transforms, include > paths, env variables, everything should be explicitly annotated as a > dependency to a target. > > Of course this is work done by a tool outside of the compiler, but it > means that the compiler should be able to be pointed to the right things > explicitly, and fail if they aren't there. Eg, include paths could be > "logical" (relative to the build system's view of the world) to support > things like remote build execution or distributed caches without a lot of > edge-casing. I've found myself vendoring and rewriting paths in libraries > to make them work with a prototype build system for the BEAM that follows > the philosophy mentioned above. > > Again, sorry I couldn't get involved in the discussion earlier, and I > hope you find this input useful! 🙌🏽. > > Side-note: I've been prototyping such a tool in the Zap project ( > https://github.com/AbstractMachinesLab/zap) in the hopes of showcasing > it in our next meeting but I haven't been able to implement the last 20% > that actually gets us the "O(1) builds". > > On Mon, Mar 1, 2021 at 10:46 PM Maxim Fedorov ***@***.***> > wrote: > >> Just looking at other build systems, they don't scan project root to >> find all applications. >> Instead, they start with some target. For example, a release (or target >> system). That release has dependencies on several applications. Build tool >> does not need to scan directories to find $app.app files, if paths are >> known. >> >> However requiring all applications to be in a single directory is not >> desirable. We have short of 400 apps, structured in a dozen sub-folders >> (some approximation of Java packages again). >> >> — >> You are receiving this because you were mentioned. >> Reply to this email directly, view it on GitHub >> <#16 (reply in thread)>, >> or unsubscribe >> <https://github.com/notifications/unsubscribe-auth/AAGQRTTUDVHQEVRJAAS5GN3TBQDKVANCNFSM4XE7ZHTQ> >> . >> > > > -- > / Leandro > -- / Leandro

-- / Leandro

[max-au]

Does it mean there is a need in another abstraction similar to "targets" in bazel/buck/...) ?
Currently I would model rebar.config as a target with multiple goals (releases defined in relx section), where every release depends on applications.
$app.app.src contain targets similar to rebar.config - rebar3 can use runtime {application, [...]} to figure out build time dependencies.
But missing mapping between applications and file system directories (and later, same issue with behaviours/parse_transforms/include_libs) makes this approach hardly implementable (especially considering various environment variables like ERL_LIBS, ERL_FLAGS, ERL_ZFLAGS, ... - which are also dependencies of the build system).

[ferd]

The runtime and build time dependencies are unrelated sets that may or may not overlap.

Using one for the purpose of the other tends to cause more issues than it solves. You can sort of get away by using the union of both sets but that tends to give less optimal build artifacts where you include compiler libs used for macros in runtime, or require a lib that's needed at runtime to be present to build a thing.

[leostera]

Does it mean there is a need in another abstraction similar to "targets" in bazel/buck/...) ?

I think so.

The way I'm thinking about the problem includes:

📏 rules: how stuff gets built, is it a library, an otp application, is it a release, is it an elixir script?

🎯 targets: named collections of files that use a rule, like a specific erlang library, or a specific OTP application spec

📦 packages: collections of targets that are meant to represent a logical unit of something within a project (like //net/http being a bunch of http libraries, apps, etc)

🏷 labels: this is the name that a package and a target have, and syntactically is //file/path/to/package:target_name

🛠 toolchains: this is the underlying compilers and other low level utilities we need to use

With that vocabulary in mind, we'd have specific rules for:

• running a parse transform
• bundling an OTP application
• testing a library
• even compiling erl -> beam
• ...

This set of rules will embody the things the current tooling can do. Each rule takes only explicit arguments.

{library, #{ name => my_lib,
             parse_transforms => ["//3rdparty/beam/lager:parse_transform"],
             srcs => ["*.erl"] }}.

They will also:

• make the process less magic
• spell out exactly what is needed when
• allow us to scale up and out
• allow us to cheaply experiment with new rules (like a container_image of snapshot_test rule)

---

So your existing project will go from a few applications to:

• a collection of packages structured however it makes the most sense for your domain or project

• that contain targets like interfaces, behaviors, models, CRUD logic, or just a single target with all your modules

• and may contain some release targets too!

And we can absolutely help bootstrap the necessary supporting files with some file-scans. Think rebar4 import . doing the scans once.

---

Why mention toolchains: it'd be fantastic if we didn't have to do this work again in mix 🙌🏽, so a way of defining pluggable compilers and making rules depend on them would allow us to introduce more rules (gleam_library, elixir_script), and reuse them in larger build graphs.

Toolchains can also be used to swap the BEAM implementation with forks or custom builds, or add support for Lumen, LAM, and other AOT compilers that'd spit out a binary.

[michalmuskala]

I do believe that fundamentally the split between built-time dependencies in rebar.config and runtime dependencies in .app.src leads only to confusion and subtle bugs when things are out-of sync. While there are cases when those differ, in 90% of cases those lists are identical. I would very much prefer for .app files to be entirely generated from rebar.config keeping all the configuration - be it build-time or runtime, in one place.

[ferd]

I want to say something on "While there are cases when those differ, in 90% of cases those lists are identical."
I think this is not the right argument. One of the most obvious cases is every time you want to build a small escript for management purposes (i.e. "interact with the system and tell me who's online") which suddenly balloon to a 250mb zip file that includes the entire prod system with it because they were declared in the same config file. My point is that you always end up having to specify whether they're the same or not; the argument you're making here is likely that we should default to them being the same, not that we shouldn't have distinct mechanisms, right? Tristan and I had argued in the past about this but dropping the files would have messed with other tooling that was out there at the time when rebar3 wasn't in as much of a dominant position. Changing it now would require breaking compatibility again, but may be doable.

One thing I'd like to point out is that we purposefully avoided designing "everything works from paths on disk" (//file/path/to/package:target_name) as a mechanism because at the point in time we re-designed rebar, it was extremely common for people to just have random libraries scattered around their local development (and prod) environments, and expecting everyone doing OSS work to also just go around and manually install stuff around.

Removing the need to specify files on disk was a concerted attempt to steer people into making packages that were requiring the same blockers to build locally as they would be requiring from other developers adding them to their project from the community. It causes pain points for things like "I can't easily import an app from an umbrella repository I have private code in" specifically because this pain point would remain invisible until someone else you publish your app for online would find themselves having the same issue down the road. Also developing with disk paths while needing to deal with dependency version management works well for monorepos but is a giant pain in the ass for OSS work only rivalled by virtual environments having to be juggled.

As a sidenote, Rebar3 played a role as a community forcing function to clean shit up, the same way hex.pm not having a namespace was a positive force (as we had hoped) to make people maintaining database drivers to either commit to forking or collaborating on bringing fixes together (I'm glad we're no longer asking "which of the 13 epgsql forks should we use?"). We possibly no longer need the same forcing functions to be present now that we re-shaped the landscape a bit (people actually version their libs now!) but I wanted to at least point out that fitness of a solution is to be contextual.

I'm not opposed to that sort of stuff where more needs to be specified by users but I wanted to give the following caveats:

• the things these large tools expect are tacit tradeoffs that most people don't necessarily like making for small projects (otherwise they'd have been using bazel already rather than rebar, rebar3, or erlang.mk which is a giant fuckyou unportable makefile) -- Rust has the same discussion in their community regarding cargo, rustc, and integration into larger build systems
• We should remain focused on designing the compiler interface here, not reinventing the whole build tool from first principles
  • the compiler interface should be usable for various types of tools (whether they're bazel-like or rebar3-like or cargo-like or flat out makefiles) interchangeably
  • I certainly will neither have nor take the time to implement a complex tool I don't need to use on behalf of companies that don't employ me (and will instead take my 4-5 free hours a week on projects I actually have fun doing rather than a somewhat unhealthy sense of duty in carrying tools for this community). I expect Tristan's OSS time to take a dive with personal changes taking place in his life now as well. I want to make it as obvious as early as possible that if expectations are that rebar3 is going to rework its core mechanisms to fit new requirements (rather than just plugging in a new compiler interface), the current maintainers are unlikely going to be those pushing it forwards. I want to moderate expectations accordingly and for you to start expecting you'll need other contributors to drive this stuff.
  • I would likely appreciate if people were to step up and want to take ownership as well. Most of my involvement the last couple of years has been pushing one or two larger pieces of work yearly and then just acting as a shepherd/product manager since the product space is large for "how people use build tools" and I'd be fine moving on to working on side projects I enjoy more if anyone wants to explicitly take some of the burden.
• I understand this is a bit of requirements gathering and this discussion is normal.