This year I had the pleasure of doing a Google I/O talk with John Ryan. Under the title “Building a package in Dart”, the aim was to showcase some of aspects involved when publishing packages on pub.dev. If you’re interested you can watch the video on youtube, or embedded below:

The talk was pre-recorded, as I wasn’t near a studio I recorded my part of the talk in my garage. And when it is clean my garage is just white walls, which doesn’t make for the coolest backdrop. So I stacked a bunch of cheap cardboard boxes for use as a background.

I obviously, did this so that I could introduce by myself by saying: “Hi, I’m Jonas, I work on package delivery, not this kind of packages”, while pointing on the boxes stacked up behind me. I will confess that this is an extraordinarily lame joke! In any case, I hope you enjoyed watching the joke as much as I enjoyed making it.

I haven’t really made this kind of video before. A it was a lot of fun, and hard work. At the end of the recording I had surprisingly little voice left, how people do longer recordings I don’t know. But this experience certainly gave me a lot of respect for youtubers doing hours of scripted video.

Disclaimer: The opinions stated here are my own, not necessarily those of my employer.

When resolving dependencies for a Dart or Flutter application it is only possible to have one version of a given package. This ensures that the dependency graph is fairly lean, since each package can only be in the graph ones. So unlike npm, you won’t end up with 3 different versions of some utility package.

On the flip side, you can run into dependency conflicts. If two packages have dependency on foo, then they cannot be used together unless there a version of foo that satisfies both constraints. As most package authors are using caret-constraints (foo: ^1.2.0) this mostly becomes a problem a new major version of foo is published. For example, pkg_a might require foo: ^1.0.0, while pkg_b requires foo: ^2.0.0, effectively preventing packages pkg_a and pkg_b from being used in the same application.

There are some workarounds for these scenarios:

• Author of pkg_a can publish a new version that is compatible with foo: ^2.0.0.
• If version 1.0.0 and 2.0.0 of foo are similar, it may be possible for the author of pkg_b to publish a version compatible with both using a constraint like foo: >=1.0.0 <3.0.0.
• If an older version of pkg_b is compatible with foo: ^1.0.0, the user may choose to use an older version of pkg_b. Indeed the solver will pick an older verison of pkg_b, if allowed by dependency constraints.
• If version 1.0.0 and 2.0.0 of foo are sufficiently similar, it may be possible for the user to employ dependency_overrides to circumvent the dependency-conflict.

However, these workarounds are not always possible. The author of pkg_a might have abandoned the package. Changes between 1.0.0 and 2.0.0 of foo might be so breaking that using a version range or dependency_overrides is infeasible. It may be that there is some functionality in foo version 1.0.0 that is not in 2.0.0 and it is critical for pkg_a to work.

At some point, you may reach the conclusion that it would better to have multiple versions of foo than finding a clean way to fix the underlying issue. In such scenarios, package:vendor can help you vendor dependencies.

Vendoring Dependencies

To vendor dependencies with package:vendor you will need a vendor.yaml file next to your pubspec.yaml. The vendor.yaml file allows you to specify:

• Which package and version to vendor.
• Under what folder name a package version should be vendored.
• What import-rewrites to apply to your application code.
• What import-rewrites to apply to the vendored packages.
• Which files to vendor for a given package.

The vendor.yaml file looks as follows:

# vendor.yaml
import_rewrites:
  # Map of rewrites that should be applied to lib/, bin/ and test/
  # Example:
  #   import "package:<package>/..."
  # is rewritten to:
  #   import "package:myapp/src/third_party/<name>/lib/..."
  # where 'myapp' is the root project name.
  <package>: <name>
vendored_dependencies:
  # Specification of which packages to be vendor into:
  #   lib/src/third_party/<name>/
  <name>:
    package: <package>
    version: <version>
    import_rewrites:
      # Rewrites to be applied inside: lib/src/third_party/<name>/
      <package>: <name>
    include:
    # Glob patterns for which files to include from the package.
    # For syntax documentation see: https://pub.dev/packages/glob
    #
    # If not specified `include` will default to the following list:
    - pubspec.yaml # always renamed vendored-pubspec.yaml
    - README.md
    - LICENSE
    - CHANGELOG.md
    - lib/**
    - analysis_options.yaml

In effect, the vendor.yaml specifies the desired state after vendoring packages. When creating a vendor.yaml file it is your responsibility that the dependencies of the package you’re vendoring are satisfied. You can do this by adding them as a dependency in your pubspec.yaml, or by vendoring the dependency and applying an import_rewrite to the original vendored package.

When you’ve written a vendor.yaml you can do dart run vendor to vendor dependencies. Vendored dependencies are written to lib/src/third_party/<name>/, and their public libraries must usually be imported from lib/src/third_party/<name>/lib/<library>.dart.

Once you have vendored dependencies it is advicable that you commit your vendor.yaml and the contents of lib/src/third_party/. The dart pub vendor command will store the current state of vendored dependencies in lib/src/third_party/vendor-state.yaml. This allows dart pub vendor to detect the changes in your vendor.yaml file. Thus, when you run dart pub vendor it will not delete vendored packages that have no changed. This is important as it allows you to patch the packages you have vendored.

Hence, you can also use package:vendor, if you need to make a quick modification of a dependency. But if you patch your vendored dependencies, you should take care to run dart pub vendor --dry run next time your vendoring. This helps you check if you’re deleting the changes you’ve made.

All in all, I think package:vendor turned out as a nice solution. It’s not a fix for everything. And if the dependency you’re vendoring integrates with a lot of other packages it can quickly get messy, and you might have to vendor a lot of dependencies to get things working. Vendoring is not a solution to every problem, but it is another tool for resolving dependency conflicts, or easily patching that one dependency that doesn’t get updated anymore.

In general, we should all hope we’ll never have to vendor dependencies.

When writing servers I often find myself having an acyclic graph of minor setup tasks. These tasks often include steps such as:

• Setup credentials from instance metadata,
• Load and validate configuration,
• Connect to database and/or shared cache (redis, memcache),
• Subscribe to a message broker (pub/sub, RabbitMQ, etc),
• Start background tasks,
• Load templates from disk,
• Setup request routers and handlers,
• Listen for incoming HTTP requests,

Depending on the service many of these steps have to be done before the server starts accepting traffic. As many of the steps depend upon the output of previous steps, this often becomes a large sequential method that does one step at time.

It usually happens that some steps are completely unrelated and could easily run concurrently. For example, there is no reason my server application can’t load templates from disk while also initiating a connection to database.

However, a method that concurrently executes unrelated steps, while ensuring that steps with inter-dependencies are executed sequentially, can quickly become complex and hard to maintain. To solve this problem I’ve published package:acyclic_steps. I’m sure there are other ways of solving this problem, feel free to show how you do this in the comment section.

Disclaimer: package:acyclic_steps is not an officially supported Google product. See the public release process documentation for details. Essentially, I enjoy publishing neat re-usable bits and patterns through the dart-neats project, which I started in order to publish neat things 🙂

At the high-level package:acyclic_steps facilities the definition of steps. Where a step may produce a value, and may have dependencies upon other steps.

/// A Step that loads configuration.
// NOTE: Typing Step<Config> is necessary for inference 
final Step<Config> configStep = Step
    .define('load-config')
    .build(() async {
  // Load configuration somehow
  return Config.fromJson(json.decode(
    await File('config.json').readAsString(),
  ));
});

/// A step that connects to a database.
final Step<DBConnection> databaseStep = Step
  .define('connect-database')
  .dep(configStep) // Add dependency upon the configStep
  .build((
  // Result from the configStep is injected as cfg.
  // Typing not needed, as Config will be inferred from configStep.
  cfg,
) async {
  return await DBConnection.connect(cfg.database);
});

On its own a step isn’t a very interesting object. You can get its name and direct dependencies, which is useful for printing the graph (and debugging). But one cannot simply execute a step.

Instead a step is executed using a Runner. The Runner takes care of executing dependent steps first, with maximum concurrency. The Runner object also caches the result of a step, such that a step is only executed once (by a given Runner), even if multiple other steps depend upon it.

Future<void> main() async {
  // Create Runner object
  final runner = Runner();

  // Run the databaseStep, which will run the configStep first.
  final dbconn = await runner.run(databaseStep);
  // NOTE: The type of the result is inferred from databaseStep.
  assert(dbconn is DBConnection);
}

Having to define a graph of steps from scratch whenever the initial input changes is not very convenient. To facilitate injection of some initial input for a graph of steps, the Runner object allows for steps to be overridden.

When a step is overridden a value is injected into the Runner objects internal cache. Such that any step that depends on the overridden step gets the injected value. This mechanism can be used to inject an initial value into a graph of steps, by defining virtual steps that must be overwritten with a value.

/// A virtual step for injecting initial configuration.
final Step<Config> configStep = Step
    .define('config')
    .build(() async {
  throw UnsupportedError('configStep must be overridden');
});

Future<void> main() async {
  // Create Runner object
  final runner = Runner();

  runner.override(configStep, Config(/* ... */));

  // Run the databaseStep, which will use the result of the overridden configStep
  final dbconn = await runner.run(databaseStep);
}

The ability to override a step can also be used for dependency injection. When writing tests it might be desirable to override the step that provides a shared cache (like redis or memcached) with a fake in-memory implementation suitable for local testing.

Beyond overriding steps the Runner object also allows the execution of steps to be wrapped. This can be useful for measuring and reporting the execution time of steps, handling or wrapping certain exceptions, or as illustrated below retry steps that fail due to I/O exceptions.

import 'package:retry/retry.dart';

Future<void> main() async {
  Future<T> wrapStepWithRetry<T>(
    Step<T> step,
    Future<T> Function() runStep,
  ) async {
    return await retry(() async => await runStep());
  }

  // Create Runner object
  final runner = Runner(wrapRunStep: wrapStepWithRetry);

  // Run the databaseStep, with a runner that will retry step execution.
  final dbconn = await runner.run(databaseStep);
}