1.3.0

Upgrading

• #1427 / #1373 / #1371 - fix staticcheck warnings, remove support for gobindata / go 1.16

Features:

• #1473 - nsqd: use --tls-root-ca-file in nsqauth request (thanks @intellitrend-team)
• #1470 / #1469 - nsqadmin: upgrade supported ECMA from ES5 to ES2020 (thanks @dudleycarr)
• #1468 - nsqadmin: add paused label to topic within the node view (thanks @dudleycarr)
• #1462 - nsqadmin: add admin check for topic/node thombstone endpoint (thanks @dudleycarr)
• #1434 - nsqd: add support of unix sockets for tcp, http, https listeners (thanks @telepenin)
• #1424 - nsqd: add /debug/freememory API (thanks @guozhao-coder)
• #1421 - nsqd: nicer tls-min-version help text default
• #1376 - nsqd: allow unbuffered memory chan if ephemeral or deferred
• #1380 - nsqd: use metadata struct for both marshal and unmarshal (thanks @karalabe)
• #1403 - nsqd: /info api returns more info (thanks @arshabbir)
• #1384 - nsqd: allow disabling both HTTP and HTTPS interfaces (thanks @karalabe)
• #1385 - nsqd: enable support for TLS1.3 (thanks @karalabe)
• #1372 - nsqadmin: new flag --dev-static-dir instead of debug build tag

Bugs:

• #1478 - Dockerfile: remove nsswitch.conf check (thanks @dudleycarr)
• #1467 - nsqadmin: fix counter by bounding animation steps (thanks @dudleycarr)
• #1466 - nsqadmin: fix broken graph template in nsqadmin node view (thanks @dudleycarr)
• #1455 / #1387 - update dependencies
• #1445 - nsqd: fix unsafe concurrency read in RemoveClient (thanks @gueFDF)
• #1441 - nsqd: fix panic when statsd enabled and memstats disabled with no topics (thanks @carl-reverb)
• #1428 - delete validTopicChannelNameRegex useless escape characters (thanks @sjatsh)
• #1419 - contrib: update nsqadmin.cfg.example (thanks @StellarisW)

1.2.1

Upgrading

• #1227 - bump dependencies, officially drop dep support, drop Go 1.9 support

Features:

• #1347 - nsqadmin: switch to go:embed for static assets
• #1355 / #1364 - arm64 builds (thanks @danbf)
• #1346 - nsqd: ability to skip ephemeral topics/channels in statsd output
• #1336 / #1341 / #1343 - nsqd: ability to configure auth endpoint path (thanks @tufanbarisyildirim)
• #1307 - remove Context to use stdlib context
• #1295 / #1296 - switch to GitHub Actions CI
• #1292 - nsqd: minimize allocations on message send (thanks @imxyb)
• #1289 - optimize uniq (thanks @donutloop)
• #1230 / #1232 - nsqd: ability to omit memory stats from /stats (thanks @creker)
• #1226 - nsqd: only update internal RDY count for client when it changes (thanks @andyxning)
• #1221 / #1363 - test against more recent versions of Go
• #1209 - nsqd: bump go-diskqueue (interface change) (thanks @bitpeng)
• #1206 - prefer idiomatic sort.Ints over sort.Sort (thanks @lelenanam)
• #1197 / #1362 - Dockerfile: update Alpine base image, use /data by default
• #1178 - nsqd: configurable queue scan worker pool (thanks @andyxning)
• #1159 - nsqd: don't buffer messages when --mem-queue-size=0 (thanks @bitpeng)
• #1073 / #1297 - nsqd: support separate broadcast ports for TCP and HTTP (thanks @shyam-king)

Bugs:

• #1347 - nsqadmin: fix graphite key for ephemeral topics/channels
• #765 / #1195 / #1203 / #1205 - fix build on illumos (thanks @i-sevostyanov)
• #1333 - fix race detector tests on non-bash shells
• #1330 - fix log_level support in configuration file (thanks @edoger)
• #1319 / #1331 / #1361 - nsqd: handle SIGTERM
• #1287 - nsqadmin: fix --proxy-graphite support (thanks @fanlix)
• #1270 / #1271 - nsqlookupd: fix incorrect error message for HTTP listener (thanks @TangDH03)
• #1264 - fix benchmark script
• #1251 / #1314 / #1327 - nsqd: fix live lock for high churn ephemeral topic/channel reconnections (thanks @slayercat)
• #1237 - Dockerfile: add nsswitch.conf to ensure go resolver uses /etc/hosts first
• #1217 / #1220 - nsqd: improve error message when --data-path does not exist (thanks @mdh67899)
• #1198 / #1190 / #1262 - synchronize close of all connections on Exit (thanks @benjsto)
• #1188 / #1189 - nsqadmin: fix channel delete, fix source-maps in Firefox (thanks @avtrifanov)
• #1186 - nsqadmin: fix nodes list with ipv6 addresses (thanks @andyxning)

1.2.0

Upgrading

• #1055 - nsqd: removed support for old metadata scheme used in v0.3.8 and earlier
  • you cannot upgrade directly from v0.3.8 to v1.2.0, you must go through v1.0.0-compat or v1.1.0
• #1115 - manage dependencies with go modules
  • dep support still present for now, but deprecated

Features:

• #1136 - nsqd: add --max-channel-consumers (default unlimited) (thanks @andyxning)
• #1133 - nsqd: add --min-output-buffer-timeout (default 25ms) to limit how low a timeout a consumer can request
  • and raise default --max-output-buffer-timeout to 30 seconds (lower timeout, more cpu usage)
• #1127 - nsqd: add topic total message bytes to stats (thanks @andyxning)
• #1125 - nsqd: add flag to adjust default --output-buffer-timeout (thanks @andyxning)
• #1163 - nsqd: add random load balancing for authd requests (thanks @shenhui0509)
• #1119 - nsqd: include client TLS cert CommonName in authd requests
• #1147 - nsq_to_file: include topic/channel in most log messages
• #1117 - nsq_to_file: add --log-level and --log-prefix flags
• #1117/#1120/#1123 - nsq_to_file: big refactor, more robust file switching and syncing and error handling
• #1118 - nsqd: add param to /stats endpoint to allow skipping per-client stats (much faster if many clients)
• #1118 - nsqadmin, nsq_stat: use include_clients param for /stats for a big speedup for big clusters
• #1110 - nsq_to_file: support for separate working directory with --work-dir (thanks @mccutchen)
• #856 - nsqadmin: add --base-path flag (thanks @blinklv)
• #1072 - nsq_to_http: add --header flag (thanks @alwindoss)
• #881 - nsqd: add producer client tcp connections to stats (thanks @sparklxb)
• #1071/#1074 - nsq_to_file: new flag --sync-interval (default same as previous behavior, 30 seconds) (thanks @alpaker)

Bugs:

• #1153 - nsqd: close connections that don't send "magic" header (thanks @JoseFeng)
• #1140 - nsqd: exit on all fatal Accept() errors - restart enables better recovery for some conditions (thanks @mdh67899)
• #1140 - nsqd, nsqlookupd, nsqadmin: refactor LogLevel, general refactor to better exit on all fatal errors
• #1140 - nsqadmin: switch to using judwhite/go-svc like nsqd and nsqadmin do
• #1134 - nsqadmin: fix clients count and channel total message rate (new bugs introduced in this cycle)
• #1132 - nsqd, nsqlookupd, nsqadmin: fix http error response unreliable json serialization
• #1116 - nsqlookupd: fix orphaned ephemeral topics in registration DB
• #1109 - nsqd: fix topic message mis-counting if there are backend write errors (thanks @SwanSpouse)
• #1099 - nsqlookupd: optimize /nodes endpoint, much better for hundreds of nsqd (thanks @andyxning)
• #1085 - switch GOFLAGS to BLDFLAGS in Makefile now that GOFLAGS is automatically used by go
• #1080 - nsqadmin: eslint reported fixes/cleanups

1.1.0

Upgrading from 1.0.0-compat: Just a few backwards incompatible changes:

• #1056 - Removed the nsq_pubsub utility
• #873 - nsqd flags --msg-timeout and --statsd-interval only take duration strings
  • plain integer no longer supported
• #921 - nsqd: http /mpub endpoint binary param interprets "0" or "false" to mean text mode
  • previously any value meant to use binary mode instead of text mode - (thanks @andyxning)

The previous release, version "1.0.0-compat", was curiously-named to indicate an almost (but not quite) complete transition to a 1.0 api-stable release line. Confusingly, this follow-up release which completes the transition comes more than a year later. Because there have been a fair number of changes and improvements in the past year, an additional minor version bump seems appropriate.

Features:

• #874 - nsqd: add memory stats to http /stats response (thanks @sparklxb)
• #892 - nsqd, nsqlookupd, nsqadmin: add --log-level option (deprecating --verbose) (thanks @antihax)
• #898 - nsqd, nsqlookupd, nsqadmin: logging refactor to use log levels everywhere
• #914 - nsqadmin: X-Forwarded-User based "admin" permission (thanks @chen-anders)
• #929 - nsqd: add topic/channel filter to /stats, use in nsqadmin and nsq_stat for efficiency (thanks @andyxning)
• #936 - nsq_to_file: refactor/cleanup
• #945 - nsq_to_nsq: support multiple --topic flags (thanks @jlr52)
• #957 - nsq_tail: support multiple --topic flags (thanks @soar)
• #946 - nsqd, nsqadmin: update internal http client with new go http.Transport features (keepalives, timeouts, dualstack)
  • affects metadata/stats requests between nsqadmin, nsqd, nsqlookupd
• #954 - manage dependencies with dep (replacing gpm) (thanks @judwhite)
• #957 - multi-stage docker image build (thanks @soar)
• #996 - nsqd: better memory usage when messages have different sizes (thanks @andyxning)
• #1019 - nsqd: optimize random channel selection in queueScanLoop (thanks @vearne)
• #1025 - nsqd: buffer and spread statsd udp sends (avoid big burst of udp, less chance of loss)
• #1038 - nsqlookupd: optimize for many producers (thousands) (thanks @andyxning)
• #1050/#1053 - nsqd: new topic can be unlocked faster after creation
• #1062 - nsqadmin: update JS deps

Bugs:

• #753 - nsqadmin: fix missing channels in topic list
• #867 - to_nsq: fix divide-by-zero issue when --rate not specified (thanks @adamweiner)
• #868 - nsqd: clamp requeue timeout to range instead of dropping connection (thanks @tsholmes)
• #891 - nsqd: fix race when client subscribes to ephemeral topic or channel while it is being cleaned up (reported by @slayercat)
• #927 - nsqd: fix deflate level handling
• #934 - nsqd: fix channel shutdown flush race
• #935 - nsq_to_file: fix connection leaks when using --topic-pattern (thanks @jxskiss)
• #951 - mention docker images and binaries for additional platforms in README (thanks @DAXaholic)
• #950 - nsqlookupd: close connection when magic read fails (thanks @yc90s)
• #971 - nsqd: fix some races getting ChannelStats (thanks @daroot)
• #988 - nsqd: fix e2e timings config example, add range validation (thanks @protoss-player)
• #991 - nsq_tail: logging to stderr (only nsq messages to stdout)
• #1000 - nsq_to_http: fix http connect/request timeout flags (thanks @kamyanskiy)
• #993/#1008 - nsqd: fix possible lookupd-identify-error busy-loop (reported by @andyxning)
• #1005 - nsqadmin: fix typo "Delfate" in connection attrs list (thanks @arussellsaw)
• #1032 - nsqd: fix loading metadata with messages queued on un-paused topic with multiple channels (thanks @michaelyou)
• #1004 - nsqlookupd: exit with error when failed to listen on ports (thanks @stephens2424)
• #1068 - nsqadmin: fix html escaping for large_graph url inside javascript
• misc test suite improvements and updates (go versions, tls certs, ...)

1.1.0-rc1

See release v1.1.0

1.0.0-compat

Upgrading from 0.3.8: Numerous backwards incompatible changes:

• Deprecated nsqd features removed:
  • Pre-V1 HTTP endpoints / response format:
    • /{m,}put (use /{m,}pub)
    • /{create,delete,empty,pause,unpause}_{topic,channel} (use /{topic,channel}/<operation>)
  • --max-message-size flag (use --max-msg-size)
  • V2 protocol IDENTIFY command short_id and long_id properties (use client_id, hostname, and user_agent)
  • /stats HTTP response name property (use client_id)
• Deprecated nsqlookupd features removed:
  • Pre-V1 HTTP endpoints / response format:
    • /{create,delete}_{topic,channel} (use /{topic,channel}/<operation>)
    • /tombstone_topic_producer (use /topic/tombstone)
• Deprecated nsqadmin features removed:
  • --template-dir flag (not required, templates are compiled into binary)
  • --use-statsd-prefixes flag (use --statsd-counter-format and --statsd-gauge-format)
• nsq_stat --status-every flag (use --interval)
• --reader-opt on all binaries that had this flag (use --consumer-opt)
• nsq_to_file --gzip-compression flag (use --gzip-level)
• nsq_to_http --http-timeout and --http-timeout-ms flags (use --http-connect-timeout and --http-request-timeout)
• nsq_to_http --round-robin flag (use --mode=round-robin)
• nsq_to_http --max-backoff-duration flag (use --consumer-opt=max_backoff_duration,X)
• nsq_to_http --throttle-fraction flag (use --sample=X)
• nsq_to_nsq --max-backoff-duration flag (use --consumer-opt=max_backoff_duration,X)
• nsqd --worker-id deprecated in favor of --node-id (to be fully removed in subsequent release)

This is a compatibility release that drops a wide range of previously deprecated features (#367) while introducing some new deprecations (#844) that we intend to fully remove in a subsequent 1.0 release.

Of note, all of the pre-1.0 HTTP endpoints (and response formats) are gone. Any clients or tools that use these endpoints/response formats won't work with this release. These changes have been available since 0.2.29 (released in July of 2014). Clients wishing to forwards-compatibly upgrade can either use the new endpoints or send the following header:

Accept: application/vnd.nsq version=1.0

Also, many command line flags have been removed — in almost all cases an alternative is available with a (hopefully) more obvious name. These changes have the same affect on config file option names.

On Linux, this release will automatically migrate nsq.<worker-id>.dat named metadata files to nsq.dat in a way that allows users to seamlessly downgrade from this release back to 0.3.8, if necessary. A subsequent release will clean up these convenience symlinks and observe only nsq.dat. See the discussion in #741 and the changes #844 for more details.

Performance wise, #741 landed which significantly reduces global contention on internal message ID generation, providing a ~1.75x speed improvement on multi-topic benchmarks.

Finally, a number of minor issues were resolved spanning contributions from 9 community members! Thanks!

Features:

• #766 - use alpine base image for official Docker container (thanks @kenjones-cisco)
• #775 - nsqadmin: /config API (thanks @kenjones-cisco)
• #776 - nsqadmin, nsq_stat, nsq_to_file, nsq_to_http: HTTP client connect/request timeouts (thanks @kenjones-cisco)
• #777/#778/#783/#785 - improve test coverage (thanks @kenjones-cisco)
• #788 - to_nsq: add --rate message throttling option
• #367 - purge deprecated features (see above)
• #741 - nsqd: per-topic message IDs (multi-topic pub benchmarks up to ~1.75x faster)
• #850 - nsqd, nsqlookupd, nsqadmin: add --log-prefix option (thanks @ploxiln)
• #844 - nsqd: deprecate --worker-id for --node-id and drop ID from nsqd.dat file (thanks @ploxiln)

Bugs:

• #787 - nsqlookupd: properly close TCP connection in IOLoop (thanks @JackDrogon)
• #792 - nsqdmin: fix root CA verification (thanks @joshuarubin)
• #794 - nsq_to_file: require --topic or --topic-pattern (thanks @judwhite)
• #816/#823 - nsqadmin: fix handling of IPv6 broadcast addresses (thanks @magnetised)
• #805/#832 - nsqd: fix requeue and deferred message accounting (thanks @sdbaiguanghe)
• #532/#830 - nsqd: switch to golang/snappy to fix snappy deadlock
• #826/#831/#837/#839 - nsqd: fix default --broadcast-address and error when nsqlookupd reqs fail (thanks @ploxiln @stephensearles)
• #822/#835 - nsqd: prevent panic in binary /mpub (thanks @yangyifeng01)
• #841 - nsqadmin: allow ctrl/meta+click to open a new tab
• #843 - nsqd: check for exit before requeing

0.3.8

Upgrading from 0.3.7: Binaries contain no backwards incompatible changes.

This release fixes a critical regression in 0.3.7 that could result in message loss when attempting to cleanly shutdown nsqd by sending it a SIGTERM. The expected behavior was for it to flush messages in internal buffers to disk before exiting. See #757 and #759 for more details.

A few performance improvements landed including #743, which improves channel throughput by ~17%, and #740, which reduces garbage when reading messages from disk.

We're now stripping debug info, reducing binary size, in the official binary downloads and Windows binaries are now bundled with the appropriate .exe extension (#726 and #751).

Features:

• #743 - nsqd: remove channel messagePump
• #751 - strip debug info from binaries (thanks @ploxiln)
• #740 - nsqd: reduce garbage when reading from diskqueue (thanks @Dieterbe)

Bugs:

• #757/#759 - nsqd: properly handle SIGTERM (thanks @judwhite)
• #738 - updates for latest go-options
• #730 - nsqd: diskqueue sync count on both read/write
• #734 - nsqadmin: make rate column work without --proxy-graphite (thanks @ploxiln)
• #726 - add .exe extension to Windows binaries (thanks @ploxiln)
• #722 - nsqadmin: fix connected duration > 1hr

0.3.7

Upgrading from 0.3.6: Binaries contain no backwards incompatible changes.

This release has been built with Go 1.6.

Highlights include the various work done to reduce nsqd lock contention, significantly improving the impact of high load on the /stats endpoint, addressing issues with timeouts and failures in nsqadmin (#700, #701, #703, #709).

Thanks to @judwhite, nsqd and nsqlookupd now natively support being run as a Windows service (#718). We're also now publishing official Windows releases.

nsqd will now flock its data directory on linux, preventing two nsqd from running simultaneously pointed at the same path (#583).

On the bugfix side, the most noteworthy change is that nsqd will now correctly reset health state on a successful backend write (#671).

Features:

• #700/#701/#703/#709 - nsqd: reduce lock contention (thanks @zachbadgett @absolute8511)
• #718 - nsqd/nsqlookupd: support running as a windows service (thanks @judwhite)
• #706 - nsqd: support enabling/disabling block profile via HTTP (thanks @absolute8511)
• #710 - nsqd: support POST /debug/pprof/symbol (thanks @absolute8511)
• #662 - nsqadmin: add flags for formatting statsd keys (thanks @kesutton)
• #583 - nsqd: flock --data-path on linux
• #663 - nsqd: optimize GUID generation (thanks @ploxiln)

Bugs:

• #672 - nsqd: fix max size accounting in diskqueue (thanks @judwhite)
• #671 - nsqd: reset health on successful backend write (thanks @judwhite)
• #615 - nsqd: prevent OOM when reading from nsqlookupd peer
• #664/#666 - dist.sh/Makefile cleanup (thanks @ploxiln)

0.3.6

Upgrading from 0.3.5: Binaries contain no backwards incompatible changes.

We've adopted the Contributor Covenant 1.2 Code of Conduct (#593). Help us keep NSQ open and inclusive by reading and following this document.

We closed a few longstanding issues related to nsqadmin, namely (#323, et al.) converting it to an API and single-page app (so that it is much easier to develop), displaying fine-grained errors (#421, #657), and enabling support for --tls-required configurations (#396).

For nsqd, we added support for deferred publishing aka DPUB (#293), which allows a producer to specify a duration of time to delay initial delivery of the message. We also addressed performance issues relating to large numbers of topics/channels (#577) by removing some per-channel goroutines in favor of a centralized, periodic, garbage collection approach.

In order to provide more flexibility when deploying NSQ in dynamically orchestrated topologies, nsqd now supports the ability to configure nsqlookupd peers at runtime via HTTP (#601), eliminating the need to restart the daemon.

As part of the large nsqadmin refactoring, we took the opportunity to cleanup the internals for all of the daemon's HTTP code paths (#601, #610, #612, #641) as well as improving the test suite so that it doesn't leave around temporary files (#553).

Features:

• #593 - add code of conduct
• #323/#631/#632/#642/#421/#649/#650/#651/#652/#654 - nsqadmin: convert to API / single-page app
• #653 - nsqadmin: expand notification context
• #293 - nsqd: add deferred pub (DPUB)
• #577 - nsqd: drop per-channel queue workers in favor of centralized queue GC
• #584 - nsqlookupd: improve registration DB performance (thanks @xiaost)
• #601 - nsqd: HTTP endpoints to dynamically configure nsqlookupd peers
• #608 - nsqd: support for filtering /stats to topic/channel (thanks @chrusty)
• #601/#610/#612/#641 - improved HTTP internal routing / log HTTP requests
• #628 - nsqd: clarify help text for --e2e-processing-latency-percentile
• #640 - switch --{consumer,producer}-opt to nsq.ConfigFlag

Bugs:

• #656 - nsqadmin: update statsd prefix to stats.counters
• #421/#657 - nsqadmin: display upstream/partial errors
• #396 - nsqdamin/nsqd: support for --tls-required
• #558 - don't overwrite docker root FS
• #582 - nsqd: ignore benign EOF errors
• #587 - nsqd: GUID error handling / catch errors if GUID goes backwards (thanks @mpe)
• #586 - nsqd: fix valid range for --worker-id
• #550/#602/#617/#618/#619/#620/#622 - nsqd: fix benchmarks (thanks @Dieterbe)
• #553 - cleanup test dirs
• #600 - nsqd: enforce diskqueue min/max message size (thanks @twmb)

0.3.5

Upgrading from 0.3.4: Binaries contain no backwards incompatible changes.

This is another quick bug fix release to address the broken nsqadmin binary in the v0.3.4 distribution (see #578).

See the release notes for v0.3.3 for more info.