Skip to content

Migrate bundle state filer read off workspace-files to /workspace/export #6219

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Sankalp-Mittal wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Migrate bundle state filer read off workspace-files to /workspace/export #6219

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e9d7c31
change API endpoint
Sankalp-Mittal Aug 12, 2026
a37791d
acceptance: regenerate goldens for state-read /workspace/export migra…
Sankalp-Mittal Aug 12, 2026
301ea4b
make the comment easy to understand
Sankalp-Mittal Aug 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 0 additions & 8 deletions acceptance/bundle/state/basic/out.state.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +0,0 @@
{
"method": "GET",
"path": "/api/2.0/workspace-files/Workspace/Users/[USERNAME]/.bundle/state/default/state/STATE_FILENAME"
}
{
"method": "GET",
"path": "/api/2.0/workspace-files/Workspace/Users/[USERNAME]/.bundle/state/default/state/deployment.json"
}
8 changes: 0 additions & 8 deletions acceptance/bundle/state/force_pull_commands/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,6 @@ Deployment complete!
=== bundle summary --force-pull: remote state read

>>> [CLI] bundle summary --force-pull
{
"method": "GET",
"path": "/api/2.0/workspace-files/Workspace/Users/[USERNAME]/.bundle/force-pull-commands/default/state/STATE_FILENAME"
}

=== bundle open without --force-pull: no remote state read

Expand All @@ -27,7 +23,3 @@ Opening browser at [DATABRICKS_URL]/jobs/[NUMID]?w=[NUMID]

>>> [CLI] bundle open foo --force-pull
Opening browser at [DATABRICKS_URL]/jobs/[NUMID]?w=[NUMID]
{
"method": "GET",
"path": "/api/2.0/workspace-files/Workspace/Users/[USERNAME]/.bundle/force-pull-commands/default/state/STATE_FILENAME"
}
16 changes: 8 additions & 8 deletions acceptance/bundle/user_agent/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ MISS deploy.terraform /.well-known/databricks-config 'databricks-tf-provider/[TF
MISS deploy.terraform /.well-known/databricks-config 'databricks-tf-provider/[TF_PROVIDER_VERSION] databricks-sdk-go/[SDK_VERSION] go/1.25.8 os/[OS] cli/[CLI_VERSION] terraform/1.5.5'
MISS deploy.terraform /.well-known/databricks-config 'databricks-tf-provider/[TF_PROVIDER_VERSION] databricks-sdk-go/[SDK_VERSION] go/1.25.8 os/[OS] cli/[CLI_VERSION] terraform/1.5.5'
MISS destroy.direct /api/2.0/preview/scim/v2/Me 'cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS] cmd/bundle_destroy cmd-exec-id/[UUID] interactive/none auth/pat'
MISS destroy.direct /api/2.0/workspace-files/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/resources.json 'cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS] cmd/bundle_destroy cmd-exec-id/[UUID] interactive/none auth/pat'
MISS destroy.direct /api/2.0/workspace/export 'cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS] cmd/bundle_destroy cmd-exec-id/[UUID] interactive/none auth/pat'
MISS destroy.direct /api/2.0/workspace/get-status 'cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS] cmd/bundle_destroy cmd-exec-id/[UUID] interactive/none auth/pat'
MISS destroy.direct /api/2.0/workspace/get-status 'cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS] cmd/bundle_destroy cmd-exec-id/[UUID] interactive/none auth/pat'
OK destroy.direct /api/2.1/unity-catalog/schemas/mycatalog.myschema engine/direct
Expand All @@ -61,7 +61,7 @@ OK destroy.direct /api/2.0/workspace-files/import-file/Workspace/Users/[USERNA
OK destroy.direct /api/2.0/workspace/delete engine/direct
MISS destroy.direct /.well-known/databricks-config 'cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS]'
MISS destroy.terraform /api/2.0/preview/scim/v2/Me 'cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS] cmd/bundle_destroy cmd-exec-id/[UUID] interactive/none auth/pat'
MISS destroy.terraform /api/2.0/workspace-files/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/terraform.tfstate 'cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS] cmd/bundle_destroy cmd-exec-id/[UUID] interactive/none auth/pat'
MISS destroy.terraform /api/2.0/workspace/export 'cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS] cmd/bundle_destroy cmd-exec-id/[UUID] interactive/none auth/pat'
MISS destroy.terraform /api/2.0/workspace/get-status 'cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS] cmd/bundle_destroy cmd-exec-id/[UUID] interactive/none auth/pat'
MISS destroy.terraform /api/2.0/workspace/get-status 'cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS] cmd/bundle_destroy cmd-exec-id/[UUID] interactive/none auth/pat'
OK destroy.terraform /api/2.0/workspace/export engine/terraform
Expand Down Expand Up @@ -93,30 +93,30 @@ MISS plan.terraform /.well-known/databricks-config 'cli/[CLI_VERSION] databricks
MISS plan.terraform /.well-known/databricks-config 'databricks-tf-provider/[TF_PROVIDER_VERSION] databricks-sdk-go/[SDK_VERSION] go/1.25.8 os/[OS] cli/[CLI_VERSION] terraform/1.5.5'
MISS plan.terraform /.well-known/databricks-config 'databricks-tf-provider/[TF_PROVIDER_VERSION] databricks-sdk-go/[SDK_VERSION] go/1.25.8 os/[OS] cli/[CLI_VERSION] terraform/1.5.5'
MISS plan2.direct /api/2.0/preview/scim/v2/Me 'cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS] cmd/bundle_plan cmd-exec-id/[UUID] interactive/none auth/pat'
MISS plan2.direct /api/2.0/workspace-files/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/resources.json 'cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS] cmd/bundle_plan cmd-exec-id/[UUID] interactive/none auth/pat'
MISS plan2.direct /api/2.0/workspace/export 'cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS] cmd/bundle_plan cmd-exec-id/[UUID] interactive/none auth/pat'
MISS plan2.direct /api/2.0/workspace/get-status 'cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS] cmd/bundle_plan cmd-exec-id/[UUID] interactive/none auth/pat'
MISS plan2.direct /api/2.0/workspace/get-status 'cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS] cmd/bundle_plan cmd-exec-id/[UUID] interactive/none auth/pat'
OK plan2.direct /api/2.0/workspace-files/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/deployment.json engine/direct
OK plan2.direct /api/2.0/workspace/export engine/direct
OK plan2.direct /api/2.0/workspace/get-status engine/direct
OK plan2.direct /api/2.1/unity-catalog/schemas/mycatalog.myschema engine/direct
MISS plan2.direct /.well-known/databricks-config 'cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS]'
MISS plan2.terraform /api/2.0/preview/scim/v2/Me 'cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS] cmd/bundle_plan cmd-exec-id/[UUID] interactive/none auth/pat'
MISS plan2.terraform /api/2.0/workspace-files/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/terraform.tfstate 'cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS] cmd/bundle_plan cmd-exec-id/[UUID] interactive/none auth/pat'
MISS plan2.terraform /api/2.0/workspace/export 'cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS] cmd/bundle_plan cmd-exec-id/[UUID] interactive/none auth/pat'
MISS plan2.terraform /api/2.0/workspace/get-status 'cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS] cmd/bundle_plan cmd-exec-id/[UUID] interactive/none auth/pat'
MISS plan2.terraform /api/2.0/workspace/get-status 'cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS] cmd/bundle_plan cmd-exec-id/[UUID] interactive/none auth/pat'
OK plan2.terraform /api/2.0/workspace-files/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/deployment.json engine/terraform
OK plan2.terraform /api/2.0/workspace/export engine/terraform
OK plan2.terraform /api/2.0/workspace/get-status engine/terraform
MISS plan2.terraform /.well-known/databricks-config 'cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS]'
MISS plan2.terraform /api/2.1/unity-catalog/schemas/mycatalog.myschema 'databricks-tf-provider/[TF_PROVIDER_VERSION] databricks-sdk-go/[SDK_VERSION] go/1.25.8 os/[OS] cli/[CLI_VERSION] terraform/1.5.5 sdk/sdkv2 resource/schema auth/pat'
MISS plan2.terraform /.well-known/databricks-config 'databricks-tf-provider/[TF_PROVIDER_VERSION] databricks-sdk-go/[SDK_VERSION] go/1.25.8 os/[OS] cli/[CLI_VERSION] terraform/1.5.5'
MISS plan2.terraform /.well-known/databricks-config 'databricks-tf-provider/[TF_PROVIDER_VERSION] databricks-sdk-go/[SDK_VERSION] go/1.25.8 os/[OS] cli/[CLI_VERSION] terraform/1.5.5'
MISS run.direct /api/2.0/preview/scim/v2/Me 'cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS] cmd/bundle_run cmd-exec-id/[UUID] interactive/none auth/pat'
MISS run.direct /api/2.0/workspace-files/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/resources.json 'cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS] cmd/bundle_run cmd-exec-id/[UUID] interactive/none auth/pat'
MISS run.direct /api/2.0/workspace/export 'cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS] cmd/bundle_run cmd-exec-id/[UUID] interactive/none auth/pat'
MISS run.direct /api/2.0/workspace/get-status 'cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS] cmd/bundle_run cmd-exec-id/[UUID] interactive/none auth/pat'
MISS run.direct /api/2.0/workspace/get-status 'cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS] cmd/bundle_run cmd-exec-id/[UUID] interactive/none auth/pat'
MISS run.direct /.well-known/databricks-config 'cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS]'
MISS run.terraform /api/2.0/preview/scim/v2/Me 'cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS] cmd/bundle_run cmd-exec-id/[UUID] interactive/none auth/pat'
MISS run.terraform /api/2.0/workspace-files/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/terraform.tfstate 'cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS] cmd/bundle_run cmd-exec-id/[UUID] interactive/none auth/pat'
MISS run.terraform /api/2.0/workspace/export 'cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS] cmd/bundle_run cmd-exec-id/[UUID] interactive/none auth/pat'
MISS run.terraform /api/2.0/workspace/get-status 'cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS] cmd/bundle_run cmd-exec-id/[UUID] interactive/none auth/pat'
MISS run.terraform /api/2.0/workspace/get-status 'cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS] cmd/bundle_run cmd-exec-id/[UUID] interactive/none auth/pat'
MISS run.terraform /.well-known/databricks-config 'cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS]'
Expand Down
6 changes: 5 additions & 1 deletion acceptance/bundle/user_agent/simple/out.requests.destroy.direct.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,11 @@
]
},
"method": "GET",
"path": "/api/2.0/workspace-files/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/resources.json"
"path": "/api/2.0/workspace/export",
"q": {
"direct_download": "true",
"path": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/resources.json"
}
}
{
"headers": {
Expand Down
6 changes: 5 additions & 1 deletion acceptance/bundle/user_agent/simple/out.requests.destroy.terraform.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,11 @@
]
},
"method": "GET",
"path": "/api/2.0/workspace-files/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/terraform.tfstate"
"path": "/api/2.0/workspace/export",
"q": {
"direct_download": "true",
"path": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/terraform.tfstate"
}
}
{
"headers": {
Expand Down
12 changes: 10 additions & 2 deletions acceptance/bundle/user_agent/simple/out.requests.plan2.direct.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,11 @@
]
},
"method": "GET",
"path": "/api/2.0/workspace-files/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/resources.json"
"path": "/api/2.0/workspace/export",
"q": {
"direct_download": "true",
"path": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/resources.json"
}
}
{
"headers": {
Expand Down Expand Up @@ -49,7 +53,11 @@
]
},
"method": "GET",
"path": "/api/2.0/workspace-files/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/deployment.json"
"path": "/api/2.0/workspace/export",
"q": {
"direct_download": "true",
"path": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/deployment.json"
}
}
{
"headers": {
Expand Down
12 changes: 10 additions & 2 deletions acceptance/bundle/user_agent/simple/out.requests.plan2.terraform.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,11 @@
]
},
"method": "GET",
"path": "/api/2.0/workspace-files/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/terraform.tfstate"
"path": "/api/2.0/workspace/export",
"q": {
"direct_download": "true",
"path": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/terraform.tfstate"
}
}
{
"headers": {
Expand Down Expand Up @@ -49,7 +53,11 @@
]
},
"method": "GET",
"path": "/api/2.0/workspace-files/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/deployment.json"
"path": "/api/2.0/workspace/export",
"q": {
"direct_download": "true",
"path": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/deployment.json"
}
}
{
"headers": {
Expand Down
6 changes: 5 additions & 1 deletion acceptance/bundle/user_agent/simple/out.requests.run.direct.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,11 @@
]
},
"method": "GET",
"path": "/api/2.0/workspace-files/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/resources.json"
"path": "/api/2.0/workspace/export",
"q": {
"direct_download": "true",
"path": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/resources.json"
}
}
{
"headers": {
Expand Down
6 changes: 5 additions & 1 deletion acceptance/bundle/user_agent/simple/out.requests.run.terraform.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,11 @@
]
},
"method": "GET",
"path": "/api/2.0/workspace-files/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/terraform.tfstate"
"path": "/api/2.0/workspace/export",
"q": {
"direct_download": "true",
"path": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/terraform.tfstate"
}
}
{
"headers": {
Expand Down
15 changes: 9 additions & 6 deletions bundle/deploy/filer.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@ import (
"io/fs"
"net/http"
"net/url"
"strings"

"github.com/databricks/cli/bundle"
"github.com/databricks/cli/libs/auth"
Expand Down Expand Up @@ -50,7 +49,13 @@ func (s stateFiler) Read(ctx context.Context, path string) (io.ReadCloser, error
}

var buf bytes.Buffer
urlPath := "/api/2.0/workspace-files/" + url.PathEscape(strings.TrimLeft(absPath, "/"))
// We read via the raw apiClient.Do so the workspace-id routing header goes
// through auth.WorkspaceIDHeaders, which drops the CLI-only "none" sentinel
// (auth.WorkspaceIDNone, written by `auth login --skip-workspace`). The SDK's
// Workspace client would forward that sentinel as a literal workspace id
// unless we special-cased it; reusing the shared helper here keeps the change
// small.
urlPath := "/api/2.0/workspace/export?path=" + url.QueryEscape(absPath) + "&direct_download=true"

@shreyas-goenka shreyas-goenka Aug 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why can't we use the SDK again here? I don't follow the point about auth.WorkspaceIDHeaders. Can we make this comment clearer?

@shreyas-goenka shreyas-goenka Aug 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it because the streaming endpoint is not accessible via the SDK?

@Sankalp-Mittal Sankalp-Mittal Aug 14, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no the issue is only when the during auth login the --skip-workspace flag is passed, this causes WorkspaceID = "none" to be passed (which I could handle separately, but I wanted to keep the code change as small as possible)

@Sankalp-Mittal Sankalp-Mittal Aug 14, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can check out libs/auth/arguments.go:8 for definition of this sentinel

@Sankalp-Mittal Sankalp-Mittal Aug 14, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also modified the comment

err = s.apiClient.Do(ctx, http.MethodGet, urlPath, auth.WorkspaceIDHeaders(s.apiClient.Config), nil, nil, &buf)
if err != nil {
return nil, err
Expand All @@ -72,10 +77,8 @@ func (s stateFiler) Write(ctx context.Context, path string, reader io.Reader, mo
}

// StateFiler returns a filer.Filer that can be used to read/write state files.
// We use a custom workspace filer which uses workspace-files API to read state files.
// This API has a higher than 10 MB limits and allows to export large state files.
// We don't use the same API for read because it doesn't correct get the file content for notebooks and returns
// "File Not Found" error instead.
// Reads use the streaming /workspace/export API, which is officially supported,
// scoped, and streams state files well beyond the 10 MB JSON export limit.
func StateFiler(ctx context.Context, b *bundle.Bundle) (filer.Filer, error) {
f, err := filer.NewWorkspaceFilesClient(b.WorkspaceClient(ctx), b.Config.Workspace.StatePath)
if err != nil {
Expand Down
Loading