feat(import): add script tool for multiple hbase snapshot imports)#4606
feat(import): add script tool for multiple hbase snapshot imports)#4606tianlei2 wants to merge 1 commit into
Conversation
3086725 to
958c187
Compare
40e4a02 to
af3976d
Compare
af3976d to
1c83a18
Compare
| --serviceAccount=${SERVICE_ACCOUNT} \ | ||
| --usePublicIps=false \ | ||
| --enableSnappy=true \ | ||
| --skipRestoreStep=${SKIP_RESTORE} \ |
There was a problem hiding this comment.
This is good, but how are we passing the restorePath?
There was a problem hiding this comment.
I feel like we should have a custom restore path and the script (idempoent by adding timestamp etc) and use it for restore the path and pass it as the restorepath in every job.
Also, with this model, who cleans up the restore path? is there a way to trigger a cleanup at the end of the script? or we have a tool that can be used? We can also say its a manual step. but then this script should output something to the tune of "the snapshot was imported, please cleanup $RESTORE_PATH once validation succeeds."
There was a problem hiding this comment.
Added some output for manual cleanups and passing restore path to every job.
|
|
||
| Example for manual parallel execution: | ||
| ```bash | ||
| ./run-snapshot-import.sh 0 3 & # Run this first! |
There was a problem hiding this comment.
In this model, how does the shard1. know htat it has to wait for restore to finish from shard 0? won't it run and fail becuase restore path is empty?
There was a problem hiding this comment.
updating this to run a restore-only command first
9622477 to
f9ce742
Compare
f9ce742 to
1180427
Compare
1180427 to
e2e88b7
Compare
Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:
b/429250716
This is the second PR with the shell tool script. This a child of #4600, so the only file that needs to be reviewed is the script itself (bigtable-dataflow-parent/bigtable-beam-import/run-snapshot-import.sh). If this looks ok, I will merge this with the parent PR.
The script is originally from https://docs.google.com/document/d/1T7BQF-AYY8xGbdbhwkV7zPA11A_Fh5SRjA_7nkzeTos/edit?tab=t.twzz7rrh6jz7#heading=h.m7p3xy7m76u2