Change documentation CI to push to new S3-based cloudfront infra instead of the legacy docs infrastructure #100
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,72 @@ | ||
| #!/bin/bash | ||
|
|
||
| # Documentation Building Script | ||
| # | ||
| # This script makes the lightningstream HTML documentation ready for copying to S3 | ||
| # | ||
| # Environment Variables Required: | ||
| # - AWS_ACCESS_KEY_ID: The AWS access key ID | ||
| # - AWS_SECRET_ACCESS_KEY: The AWS secret access key | ||
| # - AWS_REGION: The AWS region where resources are located | ||
| # - AWS_S3_BUCKET_DOCS: The name of the S3 bucket for documentation | ||
| # - BUILD_PATH: The root of the lightningstream directory | ||
| # | ||
| # Usage: | ||
| # ./mkdocs.sh <MKDOCS_YAML> <VERSION> <MKDOCS_IMAGE> <BUCKET_SUBDIR> | ||
|
|
||
| set -e # Exit immediately if a command exits with a non-zero status | ||
|
|
||
| # Main script execution | ||
| if [ "$#" -lt 1 ] || [ "$#" -gt 4 ]; then | ||
| echo "Usage: $0 <MKDOCS_YAML> <VERSION> <MKDOCS_IMAGE> <BUCKET_SUBDIR>" | ||
| exit 1 | ||
| fi | ||
|
|
||
| mkdocs_file="$1" | ||
| version="$2" | ||
| image="$3" | ||
| subdir="$4" | ||
|
|
||
| publish_script="${BUILD_PATH}/.github/scripts/publish_to_s3.sh" | ||
|
|
||
| # Prep temporary output location | ||
| mkdir -p ${PWD}/output/"${version}" | ||
|
|
||
| docker run -v "${PWD}:${PWD}" $image sh -c "pip install mkdocs-swagger-ui-tag && mkdocs build -f $mkdocs_file -d ${PWD}/output/${version}" | ||
|
|
||
| latestVersion=`aws s3 ls s3://"${AWS_S3_BUCKET_DOCS}"/docs.powerdns.com/$subdir/ | awk '{print $2}' | grep -v latest | awk -F '/' '/\// {print $1}' | sort -V | tail -1` | ||
|
There was a problem hiding this comment. I know this all comes verbatim from a different repo, but this line I didn't even fire up ShellCheck for to know it'll complain. Please use $(...) instead of the legacy backtick style command substitution. |
||
|
|
||
| if [ "$latestVersion" == "" ]; then | ||
| latestVersion="0" | ||
| fi | ||
|
|
||
| echo "Publishing version $version. Latest version already in S3 is $latestVersion" | ||
|
|
||
| $publish_script ${PWD}/output/${version} $subdir/${version} | ||
|
|
||
| if (( $(echo "$latestVersion" "$version" | awk '{if ($1 < $2) print 1;}') )); then | ||
| echo "This version is newer than the latest version in S3, publishing this version to latest" | ||
| $publish_script ${PWD}/output/${version} $subdir/latest | ||
| latestVersion="$version" | ||
| fi | ||
|
|
||
| # Build versions.json | ||
| versionsData=$(echo "[]" | jq) | ||
|
|
||
| while read -r docsVersion; do | ||
| if [ "$docsVersion" != "" ] && [ "$docsVersion" != "latest" ]; then | ||
| if [ $docsVersion == $latestVersion ]; then | ||
| versionsData=$(echo $versionsData | jq ". += [{\"title\": \"${docsVersion}\", \"version\": \"${latestVersion}\", \"aliases\": [\"latest\"]}]") | ||
| else | ||
| versionsData=$(echo $versionsData | jq ". += [{\"title\": \"${docsVersion}\", \"version\": \"${docsVersion}\", \"aliases\": []}]") | ||
| fi | ||
| fi | ||
| done < <(aws s3 ls s3://"${AWS_S3_BUCKET_DOCS}"/docs.powerdns.com/$subdir/ | awk '{print $2}' | awk -F '/' '/\// {print $1}') | ||
|
Comment on lines
+56
to
+64
There was a problem hiding this comment. This seems a bit convoluted when the aws s3 ls command has an
Contributor
Author
There was a problem hiding this comment. The There was a problem hiding this comment. Strange, there's no mention of that on https://docs.aws.amazon.com/cli/latest/reference/s3/ls.html There's the option to use There was a problem hiding this comment. Oh my, it is there, but Ctrl+F "--output" won't find it, because the |
||
|
|
||
| echo ${versionsData} > ${PWD}/output/versions.json | ||
|
|
||
| $publish_script ${PWD}/output/versions.json $subdir | ||
|
|
||
| $publish_script ${PWD}/doc/html/index.html $subdir | ||
|
|
||
| exit 0 | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,103 @@ | ||
| #!/bin/bash | ||
|
|
||
| # PowerDNS Documentation Publishing Script | ||
| # | ||
| # This script uploads documentation to an S3 bucket and invalidates the CloudFront cache. This uses the AWS CLI. | ||
| # | ||
| # Environment Variables Required: | ||
| # - AWS_ACCESS_KEY_ID: The AWS access key ID | ||
| # - AWS_SECRET_ACCESS_KEY: The AWS secret access key | ||
| # - AWS_REGION: The AWS region where resources are located | ||
| # - AWS_S3_BUCKET_DOCS: The name of the S3 bucket for documentation | ||
| # - AWS_CLOUDFRONT_DISTRIBUTION_ID_DOCS: The CloudFront distribution ID | ||
| # | ||
| # Usage: | ||
| # ./publish.sh <SOURCE_PATH> [TARGET_DIR] | ||
|
|
||
| set -e # Exit immediately if a command exits with a non-zero status | ||
|
|
||
| # Check if AWS CLI is installed | ||
| if ! command -v aws &> /dev/null; then | ||
| echo "AWS CLI is not installed. Please install it and try again." | ||
| exit 1 | ||
| fi | ||
|
|
||
| # Function to get content type based on file extension | ||
| get_content_type() { | ||
| case "${1##*.}" in | ||
| html) echo "text/html" ;; | ||
| css) echo "text/css" ;; | ||
| js) echo "application/javascript" ;; | ||
| json) echo "application/json" ;; | ||
| png) echo "image/png" ;; | ||
| jpg|jpeg) echo "image/jpeg" ;; | ||
| *) echo "application/octet-stream" ;; | ||
| esac | ||
| } | ||
|
|
||
| # Function to upload file or directory to S3 | ||
| upload_to_s3() { | ||
| local source_path="$1" | ||
| local dest_dir="$2" | ||
|
|
||
| if [ -d "$source_path" ]; then | ||
| for file in "$source_path"/*; do | ||
| if [ -d "$file" ]; then | ||
| upload_to_s3 "$file" "${dest_dir}/$(basename "$file")" | ||
| else | ||
| upload_file_to_s3 "$file" "${dest_dir}" | ||
| fi | ||
| done | ||
| else | ||
| upload_file_to_s3 "$source_path" "${dest_dir}" | ||
| fi | ||
| } | ||
|
|
||
| # Function to upload a single file to S3 | ||
| upload_file_to_s3() { | ||
| local file="$1" | ||
| local dest_dir="$2" | ||
| local content_type=$(get_content_type "$file") | ||
| aws s3 cp "$file" "s3://${AWS_S3_BUCKET_DOCS}/${dest_dir}/$(basename "$file")" --content-type "$content_type" || { | ||
| echo "Failed to upload $file to S3" | ||
| exit 1 | ||
| } | ||
| } | ||
|
|
||
| # Function to invalidate CloudFront cache | ||
| invalidate_cloudfront() { | ||
| local invalidation_path="$1" | ||
| aws cloudfront create-invalidation --distribution-id "${AWS_CLOUDFRONT_DISTRIBUTION_ID_DOCS}" --paths "${invalidation_path}" || { | ||
| echo "Failed to create CloudFront invalidation for ${invalidation_path}" | ||
| exit 1 | ||
| } | ||
| } | ||
|
|
||
| # Main function to publish to site | ||
| publish_to_site() { | ||
| local source_path="$1" | ||
| local target_dir="${2:-}" | ||
| local site_dir="docs.powerdns.com" | ||
|
|
||
| local full_target_dir="${site_dir}/${target_dir}" | ||
| upload_to_s3 "$source_path" "$full_target_dir" | ||
|
|
||
| local invalidation_path="/${target_dir}*" | ||
| invalidate_cloudfront "$invalidation_path" | ||
|
|
||
| echo "Published from ${source_path} to docs.powerdns.com${target_dir:+/}${target_dir}" | ||
| echo "Invalidated CloudFront cache for ${invalidation_path}" | ||
| } | ||
|
|
||
| # Main script execution | ||
| if [ "$#" -lt 1 ] || [ "$#" -gt 2 ]; then | ||
| echo "Usage: $0 <SOURCE_PATH> [TARGET_DIR]" | ||
| exit 1 | ||
| fi | ||
|
|
||
| source_path="$1" | ||
| target_dir="${2:-}" | ||
|
|
||
| publish_to_site "$source_path" "$target_dir" | ||
|
|
||
| exit 0 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I often prefer something like the following for finding my script dir, which seems to be the only reason for this required var:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This var disappears in the GH Action version anyway