Improve speed of dependencyOptimization

[NullVoxPopuli]

our dep optimization currently requires babel.

Part of this is inescapable today due to the need to handle @embroider/macros transforms.

However, in order to determine the minimal set of things to write for oxc (for use in rolldown, we must scan our entrypoints.

This immediately means that we need to traverse eagerly imported modules to find what our deps are and requires:

• gjs support -- via the templateTag() plugin
• hbs support (optionally, if you still have hbs) - -via the hbs plugin
• typescript support -- via @babel/plugin-transform-typescript
• decorator support -- via decorator-transforms
• we tell babel to use runtime helpers -- via @babel/plugin-transform-runtime
• re-add babel-plugin-ember-template-compilation because libraries cannot compile templates to wire-format since ember-source is free to change the wireformat in minor versions (it's also not public API, and we do often want to change it)

So my disabling of all embroider-provided dep-optimization plugins started empty ([]), and I gradually added more of the plugins based on error-driven development to eventually end up at a booting up once again.

my vite plugins modificatinos

  plugins: [
    analyzer({
      enabled: true,
      fileName: 'bundle.html',
      analyzerMode: 'static',
      openAnalyzer: false,
      defaultSizes: 'brotli',
    }),
    circleDependency(),
    mkcert({
      savePath: 'node_modules/.vite-plugin-mkcert/',
    }),
    icons({
      autoInstall: true,
    }),
    ember(),
    {
      async config(config) {
        console.log(config.optimizeDeps.rollupOptions.plugins);
        config.optimizeDeps.rollupOptions.plugins = [
          // hbs(),
          templateTag(),
          resolver(),
          babel({
            babelHelpers: 'runtime',
            extensions,
            configFile: false,
            plugins: [
              [
                '@babel/plugin-transform-typescript',
                {
                  allExtensions: true,
                  onlyRemoveTypeImports: true,
                  allowDeclareFields: true,
                },
              ],

              [
                'babel-plugin-ember-template-compilation',
                {
                  compilerPath: 'ember-source/dist/ember-template-compiler.js',
                  enableLegacyModules: [
                    'ember-cli-htmlbars',
                    'ember-cli-htmlbars-inline-precompile',
                    'htmlbars-inline-precompile',
                  ],
                  transforms: [...macros.templateMacros],
                },
              ],
              [
                'module:decorator-transforms',
                {
                  runtime: {
                    import: import.meta.resolve('decorator-transforms/runtime-esm'),
                  },
                },
              ],

              [
                '@babel/plugin-transform-runtime',
                {
                  absoluteRuntime: import.meta.dirname,
                  useESModules: true,
                  regenerator: false,
                },
              ],
              ...macros.babelMacros,
            ],
          }),
        ];
      },
    },
    babel({
      babelHelpers: 'runtime',
      extensions,
    }),
  ],

Unfortunately, for my app, NullVoxPopuli/limber#1968, this means that I've re-added all the babel plugins I was already using.

but my app here is very minimal, and this is likely the minimal set of transforms we need to run on an app's libraries during the depOptimization phase.

So, this is six transforms that may need to be re-implemented in oxc / as rollup plugins:

1. macros
2. gjs
3. hbs
4. typescript
5. decorator-transforms
   we won't need this when we move to spec-decorators
6. babel-plugin-ember-template-compilation

Once this is done, we can skip the whole extra parse + write from babel, and just rely on the ESTree that Rollup's transform hook provides: https://rollupjs.org/plugin-development/#transform

In the big 3 million line codebase I have at work, babel takes up about half of the build time:

Rolldown is 10-30x faster than rollup.
OXC is 40x faster than babel.

And I don't know what eliminating babel and using rollup's ASTs would be like.
I also don't know if using rollup's ASTs means we get to use OXC in rolldown.
Does rolldown still consume JS-written plugins? idk

(paired with @mansona on the creation of this list)

[NullVoxPopuli]

an idea I just had:

Is there maybe a way to change what we run the optimizeDeps babel over?
Like, we only need to run it on libraries with ember-addon in package.json#keywords (or that use @embroider/macros -- may need some tweaking) -- maybe we could continue to run esbuild whenever we exit an ember boundary -- as in, when we come across a super vanilla library, we let esbuild/rolldown handle it.

This could 100% be done with custom rollup/rolldown plugins for each of the above-mentioned transforms.
But also, I see that there is a withFilter plugin here: https://vite.dev/guide/rolldown.html#withfilter-wrapper.

This may work well for small apps with fewer dependencies, but for large apps with nearly 100 addons (i don't even know how many other libraries), it could be tricky to manage such a crazy regex for the id parameter.

[NullVoxPopuli]

I tried turning off the ts plugin in in rollupOptions' babel plugins, but its erroring an a d.mts -- which.... it should not be parsing 🙈

[NullVoxPopuli]

Here is what I'm ending up with so far in my smaller project:

So far, I've been able to:

• remove hbs() (my addons don't publish hbs)
• remove templateTag() (my addons don't publish un-transpiled gjs/gts)
• remove resolver() (I don't use loose mode for anything, gjs/gts only -- this even goes for services)
• babel is now using withFilter and only running on JS

my custom plugin

[
  {
    async config(config) {
      config.optimizeDeps.rollupOptions.plugins = [
        // hbs(),
        // templateTag(),
        // resolver(),
        withFilter(
          babel({
            babelHelpers: "runtime",
            extensions,
            configFile: false,
            plugins: [
              [
                "babel-plugin-ember-template-compilation",
                {
                  compilerPath: "ember-source/dist/ember-template-compiler.js",
                  enableLegacyModules: [
                    "ember-cli-htmlbars",
                    "ember-cli-htmlbars-inline-precompile",
                    "htmlbars-inline-precompile",
                  ],
                  transforms: [...macros.templateMacros],
                },
              ],
              // Needed until we move to spec decorators
              [
                "module:decorator-transforms",
                {
                  runtime: {
                    import: import.meta.resolve(
                      "decorator-transforms/runtime-esm",
                    ),
                  },
                },
              ],
              // needed because we use babel :(
              [
                "@babel/plugin-transform-runtime",
                {
                  absoluteRuntime: import.meta.dirname,
                  useESModules: true,
                  regenerator: false,
                },
              ],
              ...macros.babelMacros,
            ],
          }),
          { load: { id: /\.js$/ } },
        ),
      ];
    },
  },
];

Note

I need to re-introduce TS when building tests (as well as templateTag, etc)

Warning

A bunch of errors log about not being able to parse files, but it still eventually works itself out :-/

[mansona]

A bunch of errors log about not being able to parse files, but it still eventually works itself out :-/

This is because any failed bundling on the dep optimisation side will be re-tried on the Vite side, but this can cause module duplication because each of those cases will be /@fs/ prefixed along with all of their dependencies 🙈

Also any failure during dep optimisation will prevent it from creating a cache so you will never get a "warm start"

[NullVoxPopuli]

Next task for me:

• port macros to a rollup plugin, and see what's needed after that
• port babel-plugin-ember-template-compilation to a rollup plugin
• port decorator-transforms to a rollup plugin

Goal:

• eliminate babel