Use multiple .env.{environment} files

[kasperpeulen]

At this moment, the environment variables are spread all over the place.

Development

The .env.example file is checked in.

LITEFS_DIR="/litefs/data"
DATABASE_PATH="./prisma/data.db"
DATABASE_URL="file:./data.db?connection_limit=1"
CACHE_DATABASE_PATH="./other/cache.db"
SESSION_SECRET="super-duper-s3cret"
HONEYPOT_SECRET="super-duper-s3cret"
RESEND_API_KEY="re_blAh_blaHBlaHblahBLAhBlAh"
SENTRY_DSN="your-dsn"

# this is set to a random value in the Dockerfile
INTERNAL_COMMAND_TOKEN="some-made-up-token"

# the mocks and some code rely on these two being prefixed with "MOCK_"
# if they aren't then the real github api will be attempted
GITHUB_CLIENT_ID="MOCK_GITHUB_CLIENT_ID"
GITHUB_CLIENT_SECRET="MOCK_GITHUB_CLIENT_SECRET"
GITHUB_TOKEN="MOCK_GITHUB_TOKEN"
GITHUB_REDIRECT_URI="https://example.com/auth/github/callback"

# set this to false to prevent search engines from indexing the website
# default to allow indexing for seo safety
ALLOW_INDEXING="true"

# Tigris Object Storage (S3-compatible) Configuration
AWS_ACCESS_KEY_ID="mock-access-key"
AWS_SECRET_ACCESS_KEY="mock-secret-key"
AWS_REGION="auto"
AWS_ENDPOINT_URL_S3="https://fly.storage.tigris.dev"
BUCKET_NAME="mock-bucket"

These files are copied by the init script to .env which is git ignored, and the SESSION_SECRET is replaced with a random string.
For local development, those values are used.

For testing in CI, .env.example is being used.

Production

For production, none of these environment variables are used, they are set in the docker file:

ENV FLY="true"
ENV LITEFS_DIR="/litefs/data"
ENV DATABASE_FILENAME="sqlite.db"
ENV DATABASE_PATH="$LITEFS_DIR/$DATABASE_FILENAME"
ENV DATABASE_URL="file:$DATABASE_PATH"
ENV CACHE_DATABASE_FILENAME="cache.db"
ENV CACHE_DATABASE_PATH="$LITEFS_DIR/$CACHE_DATABASE_FILENAME"
ENV INTERNAL_PORT="8080"
ENV PORT="8081"
ENV NODE_ENV="production"

# Generate random value and save it to .env file which will be loaded by dotenv
RUN INTERNAL_COMMAND_TOKEN=$(openssl rand -hex 32) && \
  echo "INTERNAL_COMMAND_TOKEN=$INTERNAL_COMMAND_TOKEN" > .env

Not sure why this INTERNAL_COMMAND_TOKEN need to be stored in the .env file.
And can not be passed as environment variable to the server.

Some are set only in the build part of the docker file:

ENV COMMIT_SHA=$COMMIT_SHA
ENV SENTRY_ORG=
ENV SENTRY_PROJECT=

RUN --mount=type=secret,id=SENTRY_AUTH_TOKEN \
  export SENTRY_AUTH_TOKEN=$(cat /run/secrets/SENTRY_AUTH_TOKEN) && \
  npm run build

Staging

To override some of the variables for staging, they can be overriden in the stating deploy step:

      - name: 🚀 Deploy Staging
        if: ${{ github.ref == 'refs/heads/dev' }}
        run: |
          flyctl deploy \
            --env SENTRY_ENVIRONMENT=staging \
            --image "registry.fly.io/${{ steps.app_name.outputs.value }}-staging:${{ github.sha }}" \
            --app ${{ steps.app_name.outputs.value }}-staging

Secrets

Some secrets (not all are really secrets though) are added by fly secrets set manually (or in the init script).
The init script will set ALLOW_INDEXING=false to staging, and won't set this variable for production.

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_REGION
AWS_ENDPOINT_URL_S3
BUCKET_NAME
FLY_CONSUL_URL
HONEYPOT_SECRET
SENTRY_DSN
SESSION_SECRET
ALLOW_INDEXING

Test

For testing, environment variables are set in playwright.config.ts:

PORT=3000,
NODE_ENV="test"

The rest of the variables for testing are coming from in .env.example in CI, and .env for local testing.

Proposal

Proposal, clean this up, by using multiple .env.{environment} and .env.{environment}.local files.
This pattern became popular in Ruby on Rails in the 2010's, as part of the the Twelve-Factor App methodology (https://12factor.net/config). Later it also adopted by the JS ecosystem (CRA, Nextjs and Vite).

The advantage of using multiple files, is that you easily can see which environment variables will be used in each environment.
The epic stack has 4 environments: development (local), test (ci), staging (fly) and production (fly)

The gitignored .env.{environment}.local files allow to slightly modify the environment variables when run locally.
And add production secrets that can not be committed.

For example, the epic stack could use the following:

 ## .env.development
APP_ENV="development"
NODE_ENV="development"
LITEFS_DIR="/litefs/data"
DATABASE_PATH="./prisma/data.db"
DATABASE_URL="file:./data.db?connection_limit=1"
CACHE_DATABASE_PATH="./other/cache.db"
ALLOW_INDEXING="false"
PORT=3000
MOCKS=true
SENTRY_ENVIRONMENT="development"

## .env.development.local (.gitignored)
# you could override locally, to not use mocks in development
MOCKS=false

## .env.test -> extend .env.development
APP_ENV="test"
NODE_ENV="test"
MOCKS=true
SENTRY_ENVIRONMENT="test"
# possibly log sentry errors also in test, for easier debugging errors
SENTRY_DSN="some-dsn"
PORT=3000

## .env.production
APP_ENV="production"
NODE_ENV="production"
SENTRY_ENVIRONMENT="staging"
SENTRY_DSN="some-dsn"
LITEFS_DIR="/litefs/data"
DATABASE_FILENAME="sqlite.db"
DATABASE_PATH="/litefs/data/sqlite.db"
DATABASE_URL="file:/litefs/data/sqlite.db"
CACHE_DATABASE_FILENAME="cache.db"
CACHE_DATABASE_PATH="/litefs/data/cache.db"
INTERNAL_PORT="8080"
PORT="8081"
SENTRY_DSN="some-dsn"
ALLOW_INDEXING=true
MOCKS=false

## .env.staging -> extends .env.production
APP_ENV="staging"
NODE_ENV="production"
SENTRY_ENVIRONMENT="staging"
ALLOW_INDEXING=false
# might use other APIs in staging etc.

## .env.production.local (.gitignored)
# override database path when mimicking production locally
DATABASE_URL="file:./data.db?connection_limit=1"
CACHE_DATABASE_PATH="./other/cache.db"

## .env.local (.gitignored)
# Set your own secrets locally
SESSION_SECRET=
HONEYPOT_SECRET=
INTERNAL_COMMAND_TOKEN=
RESEND_API_KEY=
SENTRY_AUTH_TOKEN=
GITHUB_CLIENT_ID=
GITHUB_CLIENT_SECRET=
GITHUB_TOKEN=
GITHUB_REDIRECT_URI=
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_REGION=
AWS_ENDPOINT_URL_S3=
BUCKET_NAME=

## .env (not .gitignored anymore)
# Some shared non secret ENV variabls could be set here
SENTRY_ORG=
SENTRY_PROJECT=

How to implement

The dotenv package can be configured to read the APP_ENV and then run the right set of env files based on the APP_ENV.
Node also has a built in --env-file argument:
https://nodejs.org/api/cli.html
But this will not work for loading env variables in docker and github actions. Or if you want to load those env variable files into any other custom command.

This package is created by the maintainer of dotenv to solve that loading .env files for custom commands:
https://github.com/dotenvx/dotenvx

Another new package, varlock, seems more promising to me though;
https://github.com/dmno-dev/varlock

It uses the .env.{environment}.local hierachy convention out of the box.
Has good conventions for security.
And has intergrations for vite, docker and github actions.

[kentcdodds]

I'm open to this 👍 I'd look at a PR that adds varlock.