Improve docker/container image situation

[colin-stubbs]

The containering process at the moment has a very developer/bespoke server focus, e.g. everything assumes someone is cloning the git repo and building the container image locally before running server/agent/watchdog, with various things like a volume definition of ".:/app" to inject the latest code that may have been modified locally.

It'd be good to improve a few things so eventually people (including myself!) can rapidly spin up the latest instance of bbot-server without faffing about in a development environment, e.g. locally/in testing envs on Docker Desktop/Portainer, as well as remotely on AWS/GCP/K8S etc services.

From what I can tell the following is needed right now:

1. Dockerfile build improvements - e.g. build an image that truly expects to be run as a container without any special volume based injection of files or other compose based tweaks that currently make things work.
   i. Add COPY ./bbot_server/defaults_docker.yml /app/bbot_server/defaults.yml to remove the need to inject custom defaults at run time.
   ii. Modify CMD to match the compose.yml example of CMD ["bbctl", "server", "start", "--api-only", "--listen", "0.0.0.0"], e.g. the current default without explicit --listen 0.0.0.0 server listens only on 127.0.0.1/::1 so this will ensure that when bbot-server is running inside a container it will listen on any available IP, which is necessary to expose to the container host at minimum. I like that the python code itself defaults to localhost only - good practice and ideal if someone is somehow running things outside of a container - but given the containerisation situation anytime the container runs as a server instance (the default) it's going to need to listen to 0.0.0.0 or :: anyway, in order to be useful.

NOTE: I accept I may be missing some of the logic in not having that by default in Dockerfile...

2. Fix missing python coverage module requirement, e.g. add coverage = "^7.9.1" under [tool.poetry.dependencies] at minimum, which seems to be missing if building the container based on current compose.yml file.

e.g. I was getting this from the server instance with vanilla build, I suspect the volume .:/app injection is what was avoiding that during normal bbctl controlled operation that's using the existing compose.yml file.

server-1    | Traceback (most recent call last):
server-1    |   File "/usr/local/bin/bbctl", line 5, in <module>
server-1    |     from bbot_server.cli.bbctl import main
server-1    |   File "/app/bbot_server/__init__.py", line 11, in <module>
server-1    |     import coverage
server-1    | ModuleNotFoundError: No module named 'coverage'

3. Publish releases to Docker Hub similar to existing BLS public release containers, e.g. single bbot-server image at https://hub.docker.com/r/blacklanternsecurity/bbot-server

4. Have an example docker compose for server that does not require a local repo clone.

5. Have an example docker compose for agent that supports scale out, e.g. runs lots of agents to share scan load. I assume this is the purpose with having separate agent instances that register to server for scan runs anyway... Mostly this is going to mean zero requirement for injection of a config.yml file so that the agent knows the URL and API key to utilise when communicating with the server, e.g.
   i. Additional command line arg for API key, as --url already exists. AND/OR
   ii. Utilise env variables for URL and API key, e.g. BBOT_SERVER_URL and BBOT_SERVER_API_KEY ?

We could just hack around this in the meantime with the /app/bbot_server/default_agent.sh entrypoint script for agent that uses env vars to modify the config.yml file at run time.

5. Improve .gitignore with typical exclusions, e.g. it's missing .env atm... build.* would also be useful for any developer local scripting/configs/etc.

I'm happy to discuss further here, test, and build a PR based on feedback.

My first cut at it is this diff to the current repo content to fix minor issues as above,

colin.stubbs@nuc:~/SRC/GitHub/other/bbot-server$ git diff
diff --git a/.gitignore b/.gitignore
index c18dd8d..681421f 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1 +1,3 @@
 __pycache__/
+.env
+build.*
diff --git a/Dockerfile b/Dockerfile
index 67e57f1..66d6c35 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -2,8 +2,9 @@ FROM python:3.11
 RUN apt-get -y update
 RUN apt-get -y install curl jq
 COPY . /app
+COPY ./bbot_server/defaults_docker.yml /app/bbot_server/defaults.yml
 WORKDIR /app
 # install bbot_server in editable mode
 RUN pip install -e .
 EXPOSE 8807
-CMD ["bbctl", "server", "start", "--api-only"]
+CMD ["bbctl", "server", "start", "--api-only", "--listen", "0.0.0.0"]
diff --git a/pyproject.toml b/pyproject.toml
index 5f0dccf..60fb152 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -29,6 +29,7 @@ bbot = {git = "https://github.com/blacklanternsecurity/bbot", rev = "3.0"}
 click = "8.1.8"
 mcp = "1.7.0"
 jmespath = "^1.0.1"
+coverage = "^7.9.1"
 
 [tool.poetry.scripts]
 bbctl = 'bbot_server.cli.bbctl:main'
colin.stubbs@nuc:~/SRC/GitHub/other/bbot-server$ 

A bbot-server container image published to docker.io/colinstubbs/bbot-server:latest temporarily.

With this compose (it has a few more things going on that is really necessary, e.g. with explicit hostnames, but I was testing the env vars after working out they exist),

networks:
  network:
    name: ${NETWORK:-localnet}
    external: true

x-bbot-server-base: &bbot-server-base
  image: colinstubbs/bbot-server:latest
  domainname: ${DOMAINNAME:-localhost}
  restart: unless-stopped
  environment:
    - TZ=${TZ:-Australia/Brisbane}
    - BBOT_SERVER_CONFIG=${BBOT_SERVER_CONFIG:-/root/.config/bbot_server/config.yml}
    - BBOT_SERVER_EVENT_STORE_MONGO_URI=${BBOT_SERVER_EVENT_STORE_MONGO_URI:-mongodb://bbot-server-mongodb:27017/bbot_eventstore}
    - BBOT_SERVER_ASSET_STORE_MONGO_URI=${BBOT_SERVER_ASSET_STORE_MONGO_URI:-mongodb://bbot-server-mongodb:27017/bbot_assetstore}
    - BBOT_SERVER_USER_STORE_MONGO_URI=${BBOT_SERVER_USER_STORE_MONGO_URI:-mongodb://bbot-server-mongodb:27017/bbot_userstore}
    - BBOT_SERVER_REDIS_URI=${BBOT_SERVER_REDIS_URI:-redis://bbot-server-redis:6379/0}
    - BBOT_TESTING=${BBOT_TESTING:-False}
  networks:
    network:
  volumes:
    - "${CONFIG_DIR:-~/.config/bbot_server}/config.yml:/root/.config/bbot_server/config.yml"

services:
  server:
    <<: *bbot-server-base
    hostname: ${HOSTNAME_BASE:-bbot}-server
    ports:
      - "${BBOT_LISTEN_ADDRESS:-127.0.0.1}:${BBOT_PORT:-8807}:8807"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8807/v1/docs"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 30s
    depends_on:
      - mongodb
      - redis
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.bbot.rule=Host(`${HOSTNAME_BASE:-bbot}.${DOMAINNAME:-localhost}`)"
      - "traefik.http.services.bbot.loadbalancer.server.port=8807"
      - "wud.watch=true"

  watchdog:
    <<: *bbot-server-base
    hostname: ${HOSTNAME_BASE:-bbot}-server-watchdog
    command: ["bbctl", "server", "start", "--watchdog-only"]
    depends_on:
      server:
        condition: service_healthy
    labels:
      - "traefik.enable=false"
      - "wud.watch=true"

  agent:
    <<: *bbot-server-base
    hostname: ${HOSTNAME_BASE:-bbot}-server-agent
    command: ["bbctl", "agent", "start"]
    entrypoint: ["bash", "/app/bbot_server/default_agent.sh"]
    depends_on:
      server:
        condition: service_healthy
    labels:
      - "traefik.enable=false"
      - "wud.watch=true"

  mongodb:
    image: mongo:latest
    hostname: ${HOSTNAME_BASE:-bbot}-server-mongodb
    domainname: ${DOMAINNAME:-localhost}
    restart: unless-stopped
    volumes:
      - bbot_server_mongodb:/data/db
    networks:
      network:
    labels:
      - "traefik.enable=false"
      - "wud.watch=true"

  redis:
    image: redis:latest
    hostname: ${HOSTNAME_BASE:-bbot}-server-redis
    domainname: ${DOMAINNAME:-localhost}
    restart: unless-stopped
    networks:
      network:
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 5s
      retries: 3
    labels:
      - "traefik.enable=false"
      - "wud.watch=true"

volumes:
  bbot_server_mongodb:

[colin-stubbs]

bbctl seems to be OK with things being started independent of it...

colin.stubbs@nuc:~/SRC/GitHub/other/bbot-server$ bbctl server status
NAME                     IMAGE                            COMMAND                  SERVICE    CREATED             STATUS                       PORTS
bbot-server-agent-1      colinstubbs/bbot-server:latest   "bash /app/bbot_serv…"   agent      About an hour ago   Up About an hour             8807/tcp
bbot-server-mongodb-1    mongo:latest                     "docker-entrypoint.s…"   mongodb    About an hour ago   Up About an hour             27017/tcp
bbot-server-redis-1      redis:latest                     "docker-entrypoint.s…"   redis      About an hour ago   Up About an hour (healthy)   6379/tcp
bbot-server-server-1     colinstubbs/bbot-server:latest   "bbctl server start …"   server     About an hour ago   Up About an hour (healthy)   127.0.0.1:8807->8807/tcp
bbot-server-watchdog-1   colinstubbs/bbot-server:latest   "bbctl server start …"   watchdog   About an hour ago   Up About an hour             8807/tcp
colin.stubbs@nuc:~/SRC/GitHub/other/bbot-server$ 

[TheTechromancer]

@colin-stubbs thanks for working on this. It's true that the current setup is more ideal for devs than users, and I would have fixed it by now if it wasn't for an awkward chicken-and-egg between docker and pip.

The goal of course is to pipx install bbot-server, bbctl server start, and done.

The problem I ran into was that in order to publish compose.yml along with the pip package, it needs to be nested inside the bbot_server dir (stuff in the project root isn't published). But moving compose.yml prevents building the image from the local codebase, since Docker won't let you COPY the parent folder into the build environment.

Probably what we should do is set up auto publishing to Docker hub similar to how we already did for BBOT, and then have two compose files, one for devving and one that pulls from docker hub. The biggest challenge here would be setting up the docker tests to use the locally built image instead of the one from docker hub.

[TheTechromancer]

So plan of attack here would be

1. Add github workflow for docker hub publishing
2. Separate prod + dev compose files (preferably keeping shared elements in a common file)
3. Update bbctl to point to the new compose file (with option to override)
4. Update docker tests to point to dev version of compose file, so it's using the local code

As always happy to accept any contributions including the .gitignore etc.

[TrebledJ]

Like the plan of attack.

On a somewhat relevant tangent, I also want to bring to your attention that the RUN pip install -e . Dockerfile step could use improvement. Due to Docker's caching of intermediate steps, the bbot installed to the agent will be pinned at a specific commit.

This may lead to reproducibility hiccups, especially if there are breaking changes in the bbot-to-bbot-agent interface (or if simpletons like me are looking to debug certain things). This happens in situations when the bbot revision (currently 3.0) is updated, but no files were changed on bbot-server.

In an actual scenario, this may lead to an existing user attempting to grab the latest bbot updates with bbctl server down + bbctl server start (but unknowingly deploying the same bbot hash due to Docker caching); whereas a new user doing a fresh install would obtain the newest bbot updates. In these cases, users must remember to clear the cache or change a certain file so that the agent properly grabs the latest commit.

I'm curious if you've run into this in your own environment.

I understand this is a developer-focused hack suitable at the moment, but it took an insignificant amount of time to troubleshoot. :') If wanted, I can open this as a separate issue, but felt it was relevant to the OP here.

[TheTechromancer]

@TrebledJ thanks for the testing and sorry it took so much time. As you said the deployment isn't polished yet especially since BBOT 3.0 hasn't made it to dev.

I'd be open to any improvements either to the dockerfile or maybe the addition of a new command, e.g. bbctl server update, which forcefully builds with --no-cache.

[TrebledJ]

I didn't mean for my previous comment to sound like a rant, haha. I think ultimately it's a versioning issue between subproject and superproject. I don't have a strong recommendation for versioning. My only thought is that fixating bbot to unchanging revisions (like hashes or 3.0.0-alphaX tags) would be better for reproducibility at the expense of convenience and commit noise.

I understand that bbot-server is still undergoing development, so perhaps reproducibility of old commits isn't the first concern. And it's not a pressing concern for me either. But I'll leave this for your consideration, of when to transition to a slightly more "formal" versioning. :)