🧪 Project References for Stencil

This change has a lot of advantages for us but does bring some increased complexity to our build and development process. Some of the things that it encourages us to do (like avoiding circular dependencies between modules) are good things to do anyhow, but they also impose a bit of a learning curve and adopting them project-wide will require some substantial changes to how the code is architected¹. Overall I believe that it will be a helpful change for us to adopt.

What the heck is this about? --build? Project references?

It’s helpful to read the documentation from the TypeScript project about this feature. In short, project references allows defining sub-units of a larger codebase which can be compiled independently from the rest of the codebase, allowing TypeScript to act as both a compiler and a build tool.

Ok, how’s it work? Well, glad you asked!

• Within a single root project multiple sub-projects are defined, each with their own tsconfig.json.
  • Dependencies on other sub-projects have to be explicitly declared for each project via the references field in each project’s tsconfig.json.
  • Each sub-project can set its own compilerOptions in tsconfig.json and thereby set and enforce it’s own norms.
  • The dependency graph between all the subprojects must be a DAG². When you pass the --build flag tsc will enforce this rule and will not build until you get rid of the circular dependencies.
• The --build flag causes tsc to act as both a compiler and a build system.
  • Instead of just building a single project rooted in the directory where tsc is called it will construct a dependency graph of all projects within the root project.
  • It will then (I believe) walk the tree from the root out until it starts to find sub-projects which it can compile (i.e. sub-projects which do not have any un-built dependencies). These will then be built, in turn allowing the projects which depend on them to build, and so on until the whole project can be built.
• tsc --build also stores and re-uses the output for sub-projects and will only rebuild them if they have changed.
  • Most of the time this greatly speeds up builds, allowing only a small amount of code that has been changed to be rebuilt.
• Files within a given project can import from files in another project just as they would if they were all part of one single project. tsc keeps track of which files are included in each sub-project.

How is the change accomplished for Stencil in particular?

There are a few changes we need to make in order to get this working in Stencil.

Move compiler options to a shared tsconfig-base.json

Basically all of the compiler options that we want to share across projects need to be moved into a shared tsconfig.json file. This is called tsconfig-base.json, and looks like this:

{
  "compilerOptions": {
    "alwaysStrict": true,
    "allowSyntheticDefaultImports": true,
    "allowUnreachableCode": false,
    "declaration": true,
    "declarationMap": true,
    "experimentalDecorators": true,
    "forceConsistentCasingInFileNames": true,
    "jsx": "react",
    "jsxFactory": "h",
    "jsxFragmentFactory": "Fragment",
    "lib": ["dom", "es2019"],
    "module": "esnext",
    "moduleResolution": "node",
    "noImplicitAny": true,
    "noImplicitOverride": true,
    "noImplicitReturns": true,
    "noUnusedLocals": true,
    "noUnusedParameters": true,
    "outDir": "build",
    "rootDir": "src/",
    "strictFunctionTypes": true,
    "pretty": true,
    "resolveJsonModule": true,
    "sourceMap": true,
    "target": "es2018",
    "useUnknownInCatchVariables": true,
    "baseUrl": ".",
    "paths": {
      "@app-data": ["src/app-data/index.ts"],
      "@app-globals": ["src/app-globals/index.ts"],
      "@compiler-deps": ["src/compiler/sys/modules/compiler-deps.ts"],
      "@dev-server-process": ["src/dev-server/server-process.ts"],
      "@hydrate-factory": ["src/hydrate/runner/hydrate-factory.ts"],
      "@platform": ["src/client/index.ts"],
      "@runtime": ["src/runtime/index.ts"],
      "@stencil/core/compiler": ["src/compiler/index.ts"],
      "@stencil/core/declarations": ["src/declarations/index.ts"],
      "@stencil/core/dev-server": ["src/dev-server/index.ts"],
      "@stencil/core/internal": ["src/internal/index.ts"],
      "@stencil/core/internal/client": ["src/client/index.ts"],
      "@stencil/core/internal/client/patch-browser": ["src/client/client-patch-browser.ts"],
      "@stencil/core/internal/client/patch-es": ["src/client/client-patch-esm.ts"],
      "@stencil/core/internal/testing": ["src/testing/platform/index.ts"],
      "@stencil/core/mock-doc": ["src/mock-doc/index.ts"],
      "@stencil/core/testing": ["src/testing/index.ts"],
      "@sys-api-node": ["src/sys/node/index.ts"],
      "@utils": ["src/utils/index.ts"]
    }
  }
}

Then in each project we just use the extends key w/ the relative path to tsconfig-base.json in order to get these options everywhere, like this minimal example:

{
  "extends": "../tsconfig-base.json",
  "compilerOptions": {
    "composite": true
  }
}

The general idea is that across the projects things should be almost always compiled the same way.

Designate a few sub-projects

For the first PR introducing this I carved off a few things that were relatively simple to get building as sub-projects. Recall that sub-projects need to have no circular dependencies with other parts of the codebase — this means that it’s generally easier to get things working with a smaller directory which has minimal dependencies on other parts of the codebase. The best thing possible is to carve off a ‘leaf’ node which has no dependencies on other projects at all!

src/compiler/diagnostic

This is a small project which just exports the Diagnostic interface and nothing else. Since this interface does not include anything other than a big pile of strings it can be a leaf node! 🎉

The tsconfig.json file for this project is accordingly very minimal:

{ 
  "extends": "../../../tsconfig-base.json",
  "compilerOptions": {
    "composite": true,
    "strict": true
  },
  "include": [
    "./index.ts"
  ]
}

src/compiler/sys/environment

This project is also a leaf node which contains only the file currently located at src/compiler/sys/environment.ts on main.

src/compiler/sys/logger

This project contains all the code related to our Logger interfaces and helper functions and whatnot. It is not a leaf node because it depends on the environment and the diagnostic projects mentioned above, so it’s tsconfig.json (at src/compiler/sys/logger/tsconfig.json) looks like this:

{
  "extends": "../../../../tsconfig-base.json",
  "compilerOptions": {
    "composite": true,
    "strict": true
  },
  "include": [
    "./index.ts",
    "./logger.ts",
    "./console-logger.ts",
    "./terminal-logger.ts"
  ],
  "references": [
    { "path": "../environment" },
    { "path": "../../diagnostic" }
  ]
}

Other candidate projects

In addition to those listed above there are a few parts of the codebase which could (potentially!) be carved off right now without too much work:

• the interfaces and builder / factory code for CompilerSystem
• the CLI subdirectory
• the configuration validation code (this would likely need to involve moving the configuration interface definition)

There are more that will be more of a lift.

Moving interfaces and types out of centralized declaration files

In order to carve off sections of the codebase which do not have circular dependencies galore we need to move away from the centralized declaration files. Since everything depends on them (and they in turn depend on certain types exported elsewhere) the dependency graph for modules in the Stencil codebase is quite cyclical.

One example of this for the initial PR was moving the Logger interface and related types like LogLevel out of src/declaration/stencil-public-compiler.ts and into the src/compiler/sys/logger sub-project. This means that the code written in that sub-project no longer needs to have a dependency on that huge declaration file, so it can be more easily separated out as a self-contained module.

If we want to really go all the way with this strategy we’ll need to do a good deal more of this.

Advantages

• Faster builds during development
• Ability to set compiler options independently for sub-projects
  • This will allow us to carve off sections of the project where we can turn on strictNullChecks, and since the strictNullChecks errors will be reported as part of a normal build once we turn the option on for a subproject we’ll be basically ‘canonizing’ those errors as fixed³.
• The requirement that sub-project dependencies be a DAG enforces modularity.
  • This means that although we can continue to have circular dependencies between modules within a project we are not allowed to have circular dependencies between projects.

Disadvantages

• Greater build complexity
  • Instead of building a single project we build several projects. This requires some shift in the CLI commands we use (although this isn’t much more than adding --build here and there).
• The requirement that sub-project dependencies be a DAG requires substantial refactoring in several areas.
  • We need to pay attention to avoiding circular dependencies, and carving off new sections of the codebase to be built as separate sub-projects will require moving things around.