Feat/dynamic frontend#8
Open
Ironboxplus wants to merge 25 commits into
Open
Conversation
|
如果跨多个组件，请使用主要组件作为前缀，并在标题中枚举、描述中说明。 如果是破坏性变更，请在类型后添加 |
Ironboxplus
force-pushed
the
feat/dynamic-frontend
branch
6 times, most recently
from
3b720cc to
f0fc91a
Compare
Ironboxplus
force-pushed
the
feat/dynamic-frontend
branch
2 times, most recently
from
66b9ff7 to
bc14381
Compare
Ironboxplus
force-pushed
the
feat/dynamic-frontend
branch
from
bc14381 to
0bf12c5
Compare
…m#2471) * fix(driver): fix 189 & 189pc fastcopy form local storage * fix(driver): fix 189 & 189pc fastcopy form local storage * fix(driver): fix 189 & 189pc fastcopy form local storage
Optimize the 189pc driver to implement AccessToken login
Ironboxplus
force-pushed
the
feat/dynamic-frontend
branch
2 times, most recently
from
02e4430 to
acad358
Compare
* add LinearMemory * replace mmap with LinearMemory * remove unused code * add GuardedMemory; add `min_free_memoryMB` conf * add HybridCache and StreamBuffer * log * rename SizedReadWriterAt to Section * 重构 FileStream，改用 HybridCache * 重构 HybridCache，更新方法名并添加回滚功能；优化请求和流处理逻辑 * 重构 StreamSectionReader 接口，使用HybridCache * 在 NewGuardedMemory 函数中添加了对 LinearMemory 的最终化处理，以确保内存释放 * . * 优化检查逻辑 * 重命名 * 改进、重命名 * 修复 * 添加测试 * 移除HybridCacheReader并引入DynamicReadAtSeeker * 重构缓存读取逻辑，简化代码并引入ReadFromN方法 * 优化缓存配置注释并修复下载器部分大小限制逻辑 * 优化中断逻辑 * 优化下载器代码 * 修复bug * HybridCache添加多文件缓存模式 * 优化下载器并发，添加测试 * 修复bug * 重命名+注释 * . * fix(net): always cleanup downloader on interrupt * fix(net): guard chunk enqueue with context cancel * fix(net): update interrupt logic and add download interrupt test * fix(test): update concurrency limit in high concurrency test * refactor(buffer): simplify ReadAt logic * refactor(config): update memory configuration logic * . --------- Co-authored-by: Suyunmeng <Susus0175@proton.me>
…nd upload hash rework - Add link refresh capability with RefreshableRangeReader for expired download links - Implement self-healing reader for interrupted stream reconnection - Add seekable prefetch window with strict tests - Enhance hash calculation and upload logic for various stream types - Fix link expiry detection to avoid treating context cancellation as expired - Optimize link cache logic, remove SyncClosers dependency - Support all 4xx client errors in expired link check - Fix local file 'file already closed' error - Unify hash calculation progress weight to 100 - Add slow network support with adjusted timeout and retry
…and MD5 checksum - Add duplicate filename handling to ensure file name uniqueness - Add folder creation lock mechanism and retry logic - Increase MakeDir wait time for Google Drive API sync delay - Add retry mechanism for small file upload read errors - Update Put method to support MD5 checksum on seekable and non-seekable streams
…d error handling - Add permanent delete feature with recycle-bin retry logic - Expose proxy_range option in driver meta - Fix duplicate link error handling via error code 10008 detection - Fix folder delete cid JSON number deserialization error - Move offline task Remove before Transfer - Cleanup completed offline download tasks
… mechanism - Optimize task list retrieval with multi-page support and status hints - Add task limit wait mechanism to optimize offline download task processing
… etag fix - Implement streaming upload for Baidu Netdisk - Add rate limiting and retry logic for quark_open, optimize upload handling - Fix quark shard size adjustment for oversized errors - Fix file copy failure to 123pan due to incorrect etag - Fix aliyundrive upload hash calculation
…0 panic - Fix pre-create subdirectory depth from 2 to 1 to avoid deep recursion - Fix srcBasePath bug and remove redundant sleep in preCreateDirectoryTree - Add unit tests for preCreateDirTreeFn - Update storage retrieval method in alias listRoot function - Fix sftp symlink path resolution - Fix 500 panic and NaN issues in openlist driver
…rastructure - Add dynamic frontend pull from GitHub rolling release at startup - Implement frontend caching mechanism with ETag-based validation - Track rolling release by tag commit hash - Unify configurable frontend repo and release channel across build and runtime - Upgrade GitHub Actions to Node24-compatible versions (checkout v5, setup-go v6, etc.) - Update Docker workflow to support frontend version matrix builds - Optimize CI build speed with smart caching
- Add CLAUDE.md for project guidance and development instructions - Add compatibility report - Use Ironboxplus/115-sdk-go v0.2.5 fork in go.mod replace directive
…imit, dist swap lock, Go 1.26 vet - Google Drive Put: replace infinite 401 recursion with maxPutAuthRetries=2 - Google Drive MakeDir: delete mkdirLocks entry after unlock to prevent memory leak - selfHealingReadCloser: only reconnect on io.ErrUnexpectedEOF, not normal io.EOF - RefreshableRangeReader: document concurrency contract for innerReader replacement - Frontend extractTarGz: add maxExtractFileSize (500MB) and io.LimitReader - Frontend dist swap: add distSwapMu to protect rename window - Fix Go 1.26 non-constant format string vet errors across drivers
- NewOSSUploadHttpClient: ResponseHeaderTimeout 15s → 5min for uploads - Handle PartAlreadyExist (409) in retry by recovering part via ListUploadedParts instead of failing, fixes timeout-then-retry conflict with oss.Sequential()
…leStream While hashing chunk N, asynchronously prefetch chunk N+1 via RangeRead. Overlaps network I/O with hash computation, reducing total hash time. Hash values remain identical — no change to upload flow or rapid-upload logic.
Put() only called WaitLimit once at entry, but then fired 3-4 SDK requests (UploadInit ×3, UploadGetToken) without rate limiting. Under concurrent copy tasks this overwhelmed the 115 API, causing state:false/code:0 rejections on all requests including List/MakeDir. Now each SDK call in Put is individually rate-limited, matching the pattern used by List, MakeDir, Move, etc.
added 4 commits
May 17, 2026 10:33
The SDK's authRequest had no concurrency protection on token refresh. Multiple goroutines hitting 401 simultaneously would all call RefreshToken independently, causing "refresh frequently" errors and corrupting the stored tokens. v0.2.6 adds mutex with double-check pattern to ensure only one goroutine refreshes at a time.
115 API returns {state:true, data:[]} for non-existent paths instead of
an error code. SDK v0.2.8 adds ErrDataEmpty sentinel; driver.Get() maps
it to ObjectNotFound so MakeDir can create directories normally.
…iter hash truncation
Three bugs fixed:
1. Upload callback was never validated — when 115 returned {"state":false},
the error was silently ignored. Added oss.CallbackResult capture and
checkUploadCallback() validation.
2. CacheFullAndWriter for known-size files > MaxBufferLimit only cached the
first ~48MB (via cache() truncation), producing a partial SHA1 hash.
115 then rejected the upload with code=10002 "校验文件失败". Fix: when
file size exceeds MaxBufferLimit, write to a temp file instead of the
in-memory peekBuff. This fixes all drivers using CacheFullAndHash with
large non-seekable streams (115, google_drive, aliyundrive, etc).
3. Frontend fetcher blocked startup on China servers where GitHub is
unreachable. Removed blocking download, use embedded dist immediately
and let Watcher fetch rolling release in background.
While the caller uploads block N, kick off a goroutine that reads block N+1 from the source into the next HybridCache block. The next GetSectionReader picks up the prefetched block, overlapping source download with destination upload for sequential same-sized chunks (copy tasks, 115/baidu/aliyundrive sequential multipart uploads). - Prefetch is clamped to remaining file size so the final partial chunk doesn't produce a synthetic short-read. - DiscardSection drains and (when matching) consumes the prefetched block without touching the source again. - Prefetch errors surface on the next GetSectionReader call. - Cleanup is registered via file.Add so prefetch is drained before the HybridCache is freed.
Ironboxplus
force-pushed
the
feat/dynamic-frontend
branch
from
acad358 to
7995dbd
Compare
added 3 commits
May 17, 2026 15:59
Removed preCreateDirTreeFn and its method wrapper. Directory creation
is no longer batched up-front via a one-level BFS walk; instead it
happens on demand:
- top-level dst MakeDir still runs once at line 198 (early failure
detection, active dir creation)
- sub-task RunWithNextTaskCallback creates each subdir as it
recurses into it
- op.Put internally calls MakeDir(parent) before uploading
- op.MakeDir walks the parent chain so deep trees work without
pre-creation, and our cache-sync retry handles cloud-storage
eventual consistency
The merge-mode existedObjs logic (the only invariant the BFS
precreate quietly protected, alongside OpenListTeam#1898's ObjectNotFound
tolerance) is extracted into a pure, injectable function:
existingDstFilesFn(ctx, listDst, dstPath) (map[string]bool, error)
12 unit tests pin down its contract: empty dst, mixed files/dirs,
raw and wrapped ObjectNotFound (regression for OpenListTeam#1898), other List
errors, ctx cancellation before/during iteration, duplicates,
large dst, single-call sanity. The old 11 TestPreCreateDirTreeFn_*
tests are removed with the code.
…te removal JOURNAL.md: new entry covering the three coordinated changes (commits fccab33 → 7995dbd → 6b3ce57) — rebase onto upstream HybridCache, Pass 2 prefetch on hybridSectionReader, and BFS precreate deletion. CLAUDE.md: replace the now-obsolete "MaxBufferLimit truncation pitfall" section with the current HybridCache story; add new sections for the hybridSectionReader prefetch and the merge-task ObjectNotFound tolerance contract (incl. the existingDstFilesFn extraction).
The merge skip rule at copy_move.go:221 has two halves whose
combination is load-bearing:
1. existingDstFilesFn must only include FILES (not dirs) from dst
2. the call-site guard `!obj.IsDir() && existedObjs[name]` must
keep the !obj.IsDir() clause
If either half drifts (e.g. dirs leak into existedObjs, or the
IsDir guard is removed), src subdirectories matching dst dirs of
the same name would be silently skipped — every file inside them
would never get copied. That bug is invisible to "did the task
complete?" checks; only shows up as quietly missing deep files.
New tests:
- TestMergeSkipDecision_DirsAreNeverSkipped: 7-case matrix
covering every src kind × dst state combo, including the
cross-type name collisions (src dir / dst file and vice versa)
- TestMergeSkipDecision_EmptyDst: nothing skipped against empty dst
- TestMergeSkipDecision_DeepTreeRecursionContract: 3-level src
tree against partial dst, asserts correct skip/spawn at each
level — the canonical "deep files missing" regression guard
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description / 描述
Motivation and Context / 背景
Closes #XXXX
Relates to #XXXX
How Has This Been Tested? / 测试
Checklist / 检查清单
我已阅读 CONTRIBUTING 文档。
go fmtor prettier.我已使用
go fmt或 prettier 格式化提交的代码。我已为此 PR 添加了适当的标签（如无权限或需要的标签不存在，请在描述中说明，管理员将后续处理）。
我已在适当情况下使用"Request review"功能请求相关代码作者进行审查。
我已相应更新了相关仓库（若适用）。