Skip to content

Secure log shipper docs: no public services or IPs#2420

Open
kcmartin wants to merge 4 commits into
mainfrom
fix-log-shipper-no-public-access
Open

Secure log shipper docs: no public services or IPs#2420
kcmartin wants to merge 4 commits into
mainfrom
fix-log-shipper-no-public-access

Conversation

@kcmartin

@kcmartin kcmartin commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

The Log Shipper guide told users to edit fly.toml and keep a [[services]] section on internal_port = 8686. That port serves Vector's GraphQL API (including an unauthenticated /playground), so following the docs exposed it publicly. This was reported on the community forum, where Fly staff recommended deploying with --no-public-ips.

Vector pulls logs from Fly.io's internal log stream and doesn't need to accept inbound connections, so this:

  • Removes the [[services]] / port 8686 block from the setup steps
  • Deploys with fly deploy --no-public-ips
  • Notes how to release IPs on already-exposed deployments (fly ips list / fly ips release)
  • Adds a health check section using a top-level [checks] block against Vector's /health on 8686, which works without a public service or IP (the shipper can stop shipping logs without exiting, see fix: add health check to exporting logs documentation #1828)

All of the above was verified on a live deploy; see the testing comments below.

Supersedes #930 (same fix, but that PR targeted a since-moved file and had gone stale) and #1828 (same health check goal, but via a public [http_service], which this PR removes).

The log shipper guide told users to keep a [[services]] section on
port 8686, which exposed Vector's unauthenticated GraphQL API publicly
(reported at community.fly.io/t/14836). Vector pulls logs from the
internal stream and needs no inbound access, so remove the service
section and deploy with --no-public-ips. Also note how to release IPs
on already-exposed deployments.

Supersedes #930.
@kcmartin

kcmartin commented Jul 2, 2026

Copy link
Copy Markdown
Contributor Author

Tested this end to end against a real deployment (throwaway app in my personal org, destroyed after):

  • Followed the updated instructions verbatim: launched the shipper image, deleted the generated [http_service] section, deployed with fly deploy --no-public-ips. Result: no public IPs allocated, no DNS record created, port 8686 unreachable. flyctl confirms: "Your app is deployed but does not have a public or private IP address."
  • Logs confirmed Vector connects outbound to the internal NATS stream, so the shipper genuinely needs no inbound services.
  • Negative control: redeployed with the old-style public service config and allocated IPs. Vector's GraphQL playground was served publicly at https://<app>.fly.dev/playground, reproducing the exposure from the original community report.
  • Remediation path: fly ips release on both addresses cleared the DNS records immediately; the hostname stopped resolving entirely within a few minutes.

Two things surfaced during testing and were folded into the PR: fly launch generates [http_service] (single brackets), so the text now matches, and Datadog was added to the Vale vocabulary to clear a pre-existing lint failure on this file.

@kcmartin

kcmartin commented Jul 2, 2026

Copy link
Copy Markdown
Contributor Author

DONE: #1828 raises a related need (the shipper can wedge silently, and a health check was recommended by support). Since this PR removes the public service, the right approach is a top-level [checks] machine check against Vector's /health on 8686, which needs no service or public IPs. Plan is to add a short health check section here before this merges.

@kcmartin

kcmartin commented Jul 6, 2026

Copy link
Copy Markdown
Contributor Author

Health check section added, using a top-level [checks] block against Vector's /health on 8686, so no public service or IP is needed. Verified on a live deploy: the check passes on a healthy shipper, goes critical when Vector hangs (simulated with SIGSTOP), and recovers. This covers the concern from #1828. Ready for review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant