feat: Add flag for import existing resources

[electrofelix]

Pass through option for importing existing resources which allows using
Retain for DeletionPolicy on resources to avoid removal on data
resources if stack needs to be deleted and recreated.

Fixes: #1699

[electrofelix]

Main use case I have for this is that sometimes the stack gets into a funk and needs to be deleted and recreated, however if there are data resources in the same application stack specifying DeletionPolicy: Retain prevents re-deploying via sam.

The workarounds are to move data resources out to separate stacks, which can work for event buses and DBs, but is a pain when working with queues. End up needing to pass a multitude of resource identifiers in via parameters.

Using this approach, I can mark queues, event buses (+ archives), and any DBs to be retained and be able to teardown and redeploy without dropping messages/events/data, and at the same time avoid needing a complex orchastrated deployment spanning multiple repositories or needing complex CI jobs to handle what stacks to delete vs deploy.

Note: when testing I noticed CloudFormation doesn't handle automatically re-importing AWS::Logs::LogGroup resources particular well.

I have a fairly rough script that does the same as this code the AWS CLI, which encountered the same issue suggesting there may be a bug in Cloudformation. Appears it fails to remove some information from resources associated with the original stack meaning they still appear to be part of a stack and it refuses to import. Solving this in Cloudformation would automatically fix it for this feature as well.

It would be nicer for it to be available built in rather than needing to run the script sometimes:

TEMP_TEMPLATE=$(mktemp -t template.yaml) || exit 1

stack_name=$(toml get --toml-path ${configName} ${sectionName}.global.parameters.stack_name)
capabilities=$(toml get --toml-path ${configName} ${sectionName}.global.parameters.capabilities)
parameter_overrides=$(toml get --toml-path ${configName} ${sectionName}.global.parameters.parameter_overrides | sed -e 's/^\[//g; s/\]$//g')
params=""
for param in ${parameter_overrides}
do
    param=${param%,}
    param=${param%\'}
    param=${param#\'}
    key=${param%%=*}
    value=${param#*=}
    params="${params}${params:+ }ParameterKey=${key},ParameterValue=${value}"
done

# make sure all aws commands use the requested profile
export AWS_PROFILE=$profile

echo "Packaging stack and then using cloudformation to import existing using config file: '$configName' ..."
command="sam package --config-file $configName --output-template-file ${TEMP_TEMPLATE}"
echo "Executing $command"
$command


command="aws cloudformation describe-stacks --stack-name "${stack_name}""
echo "Checking if stack exists"
echo "Executing $command"
if ! $command >/dev/null 2>&1; then
    change_set_type=CREATE
    stack_complete_command=stack-create-complete
else
    change_set_type=UPDATE
    stack_complete_command=stack-update-complete
fi

change_set_name="change-set-$(date +%s)"
command="aws cloudformation create-change-set --stack-name ${stack_name} --change-set-type ${change_set_type} \
 --change-set-name ${change_set_name} --template-body "file://${TEMP_TEMPLATE}" --capabilities ${capabilities} \
 --parameters ${params} --import-existing-resources"
echo "Executing $command"
$command

sleep 2
echo "Waiting for changeset to be complete"
command="aws cloudformation wait change-set-create-complete --stack-name ${stack_name} --change-set-name ${change_set_name}"
echo "Executing $command"
$command
if [[ $? -ne 0 ]]; then
    failure_reason=$(aws cloudformation describe-change-set --stack-name ${stack_name} --change-set-name ${change_set_name} --query "StatusReason" --output text)
    if [[ "${failure_reason}" == *"The submitted information didn't contain changes."* ]] || [[ "${failure_reason}" == *"No updates are to be performed."* ]]; then
        echo "No changes detected, exiting"
        exit 0
    else
        echo "Changeset failed to create: ${failure_reason}"
        exit 1
    fi
fi
exit 0

[vicheey]

Thank you for your contribution, @electrofelix. The current code implementation is clear and concise. We just need to verify both forward and backward deployment scenarios that this update would not leave a stack in an unrecoverable state if deployment fail. Would you be able to enhance this further by adding integration tests to cover your specific deployment use case as well?

[electrofelix]

@vicheey sure, I presume it's just follow the existing integration tests such as https://github.com/aws/aws-sam-cli/blob/develop/tests/integration/deploy/test_deploy_command.py#L1542-L1593 going with the following steps:

• deploy template with some resources marked retain (eventbus + archive, sqs, & dynamodb)
• delete stack
• re-deploy same template and assert expected resources have import in deploy_process_execute.stdout.strip()

Presumably some work around tear will be needed for this scenario? Extend add_left_over_resources_from_stack to cover the additional resource types added?

Is this better as a separate test file to cover the import functionality under the deploy tests?

Might be a few weeks before I will get back to updating

[vicheey]

Do you mind help with the make pr failure?

[electrofelix]

Do you mind help with the make pr failure?

Not at all, just delayed getting back to it

[c-ameron]

Thanks so much for this @electrofelix !

@vicheey is this up for consideration? This would help me a lot move from terraform to SAM for some resources.

Thanks!

[vicheey]

Hi @electrofelix,

I've pushed a small follow-up commit on top of your work:

• Fixed a minor typo in the --import-existing-resources help text (missing word "resources")
• Added an integration test that creates a pre-existing SSM parameter, deploys with --import-existing-resources, and verifies the changeset shows the Import action
• Added the import_existing_resources parameter to the integration test base helper
  CI is running now. Will follow up once results are in.

@c-ameron,
We're working on getting this merged. I'll need another team member to review and hopefully get this out soon.
Stay tuned!

[vicheey]

The integration test: https://github.com/aws/aws-sam-cli/actions/runs/23670588901

[seshubaws]

can we add a test for reading this param from the samconfig?

[reedham-aws]

While the idea of the PR is solid, I think there's a hidden issue with how this will interact with CloudFormation. In the past, the reason we've had difficulty implementing this is because the CloudFormation import feature does not support AWS::Serverless resources. This might have changed, but I think it's because the import step is happening before the transform is server-side. The reason the integration tests are passing is because it's not using a AWS::Serverless resource.

We do this resource import in the AWS Toolkit, but when we do it there we actually deploy the stack as raw CloudFormation before translating it back to SAM. I don't think that would work well in this case considering we're already coming with a user-defined template.

That being said, I think there needs to be changes to warn that this won't work or to not allow it all for serverless resources.

[reedham-aws]

I think in other places in the code, we don't have the template inline, but rather in tests/integration/testdata