Fsync atomic_open temp file before rename (#69583)#69584
Conversation
|
CI red is the inherited 3006.x ZMQ cluster from 36e914a ( |
salt.utils.atomicfile._AtomicWFile.close() renamed the temp file over the destination without first flushing user-space buffers and fsyncing the file's data and metadata. POSIX does not require the kernel to implicitly fsync on rename, and the fsync-on-rename heuristic of ext4 and friends is not universal. As a result a crash after the rename could expose a written-but-unsynced file (truncated or partial), defeating the durability contract callers expect from an "atomic write" helper. Flush + os.fsync() the underlying file descriptor before close() so the new file's contents are durable on disk before the rename swaps it into place. Fixes saltstack#69583
b5b43c5 to
608d6bd
Compare
What does this PR do?
Make
salt.utils.atomicfile.atomic_openactually crash-safe: flush andfsync()the temp file's data and metadata before the atomic renamethat swaps it into place over the destination.
What issues does this PR fix or reference?
Fixes #69583
Previous Behavior
_AtomicWFile.close()calledself._fh.close()and thenatomic_rename(...)with no
fsyncin between. POSIX does not require the kernel to fsyncimplicitly on rename, and the fsync-on-rename heuristic implemented by ext4
and friends is not universal. A crash after the rename could expose a
written-but-unsynced file (truncated or partial) — defeating the durability
contract callers expect from an "atomic write" helper. Users have observed
truncated files in
/etcas a consequence.New Behavior
_AtomicWFile.close()now performsself._fh.flush()andos.fsync(self._fh.fileno())before closing the temp file handle, so thenew file's contents are durable on disk before the rename swaps it into
place. The fsync call is guarded against file-like objects that don't
expose a real fd, so behavior is unchanged for callers wrapping in-memory
streams.
Merge requirements satisfied?
"atomic write" contract)
changelog/69583.fixed.md)tests/pytests/unit/utils/test_atomicfile.py::test_atomicfile_fsyncs_before_rename)Commits signed with GPG?
Yes