Add new scripts for building and cleaning wheels

release/README.md

@@ -0,0 +1,173 @@
+# Tools for building releases of TensorFlow Quantum
+
+This directory contains configurations and scripts that the TensorFlow Quantum
+maintainers use to create Python packages for software releases. The process of
+making a TFQ release is complex and has not been fully automated. The scripts
+in this directory help automate some steps and are a way of capturing the
+process more precisely, but there are still manual steps involved.
+
+## Background: how TensorFlow Quantum is linked with TensorFlow
+
+TFQ is implemented as a Python library that integrates static C++ objects. Those
+C++ objects are linked with TensorFlow static objects when both TFQ and
+TensorFlow are installed on your system. Unlike a pure Python library, the
+result is platform-dependent: the Python code itself remains portable, but the
+underlying C++ objects need to be compiled specifically for each target
+environment (operating system and CPU architecture).
+
+TensorFlow does not provide ABI stability guarantees between versions of
+TensorFlow. In order to avoid the need for users to compile the TFQ source code
+themselves when they want to install TFQ, each release of TFQ must be pinned to
+a specific version of TensorFlow. As a consequence, TFQ releases will not work
+with any other version of TensorFlow than the one they are pinned to.
+
+Python wheels for TFQ are produced by compiling them with a toolchain that
+matches that used to build the version of TensorFlow being targeted by a given
+version of TFQ. A number of elements affect whether the whole process succeeds
+and the resulting wheel is portable to environments other than the specific
+computer TFQ is built on, including:
+
+*   The version of Python and the local Python environment
+*   The version of TensorFlow
+*   The TensorFlow build container used
+*   The Crosstool configuration used
+*   Whether CUDA is being used, and its version
+*   The dependency requirements implied by Cirq, TF-Keras, NumPy, Protobuf, and
+    other Python packages
+
+## Procedure
+
+Building a TensorFlow Quantum release for Linux involves some additional steps
+beyond just building TFQ and producing an initial Python wheel. The procedure
+uses `auditwheel` to "repair" the resulting wheel; this improves the
+compatibility of the wheel so that it can run on a wider range of Linux
+distributions, even if those distributions have different versions of system
+libraries.
+
+### Preliminary steps
+
+1.  Make sure you have `pyenv`, `pip`, and `jq` installed on your system.
+
+2.  (Optional) Preinstall Python versions 3.9, 3.10, 3.11, and 3.12 into `pyenv`
+    so that `build_release.sh` can create virtual environments with those Python
+    versions without having to install the requested version(s) itself.
+
+3.  Git clone the TensorFlow Quantum repo to a directory on your computer.
+
+4.  `cd` into your local clone directory in a Bash shell.
+
+### Build the release
+
+1.  Run `./release/build_release.sh X.Y.Z`, where _X.Y.Z_ is a Python version
+    for which you want to build a TFQ release.
+
+2.  If the previous step completes successfully, proceed to the next section
+    below and test the wheel.
+
+3.  Repeat steps 1–2 above for other Python versions.
+
+### Testing the release
+
+Testing is currently not automated to the degree that building a release is.
+The following is the current process.
+
+1.  First, perform a quick local test.
+
+    1.  `cd` out of the TFQ source directory. This is essential, because
+        importing TFQ into a Python interpreter when the current directory is
+        the TFQ source tree will result in baffling errors (usually something
+        about `pauli_sum_pb2` not found).
+
+    1.  Create a fresh Python virtual environment.
+
+    1.  Run `pip install /path/to/wheel`, where `/path/to/wheel` is the path to
+        the wheel file corresponding to the version of Python you are running.
+
+    1.  (Currently required because TFQ requires the legacy version of Keras.)
+        Set the environment variable `TF_USE_LEGACY_KERAS` to `1` before
+        loading any TensorFlow or TensorFlow Quantum code. You can set it in
+        your shell before starting Python; alternatively, you can execute the
+        following Python code before any `import tensorflow` or `import
+        tensorflow_quantum` statements:
+
+        ```python
+        import os
+        os.environ["TF_USE_LEGACY_KERAS"] = "1"
+        ```
+
+    1.  Run some example TFQ code. If it runs without problems, go on to the
+        next step; if it fails, debug the error.
+
+2.  Second, test in Colab.
+
+    1.  Go to a remotely hosted Colab and make a copy of the [Hello Many Worlds
+        tutorial notebook](
+        https://www.tensorflow.org/quantum/tutorials/hello_many_worlds).
+
+    1.  Using the Colab file explorer, upload a TFQ wheel you created matching
+        the version of Python running in Colab. (At the time of this writing,
+        Colab uses Python 3.12.)
+
+    1.  When the upload finishes, right-click on the file name in the Colab file
+        explorer and copy the path to the file in Colab.
+
+    1.  Find the notebook cell that contains the `!pip install` command for
+        TensorFlow Quantum. **Replace that command** with the following, pasting
+        in the path that you copied in the previous step:
+
+        ```python
+        !pip install /here/paste/the/path/to/the/wheel/file
+        ```
+
+    1.  Run the notebook step by step. If Colab asks you to restart the session,
+        do so, and after it finishes restarting, continue with the remaining
+        cells in the notebook.
+
+    1.  If the notebook executes all the way through without error,
+        congratulations! If something fails, proceed to debug the problem.
+
+## More information
+
+"TensorFlow SIG Build" is a community group dedicated to the TensorFlow build
+process. This repository is a showcase of resources, guides, tools, and builds
+contributed by the community, for the community. The following resources may be
+useful when trying to figure out how to make this all work.
+
+*   [The "TF SIG Build Dockerfiles" README file](
+    https://github.com/tensorflow/build/tree/ff4320fee2cf48568ebd2f476d7714438bfa0bee/tf_sig_build_dockerfiles#readme)
+
+*   Other info in the SIG Build repo: https://github.com/tensorflow/build
+
+The script `build_release.sh` relies on other scripts to do the main work.
+Those steps can be run manually, and sometimes it is useful to do so when
+debugging problems. The steps are:
+
+1.  `cd` to the top level of your git clone of the TensorFlow Quantum repo.
+
+1.  Create a Python virtual environment for one of the supported versions of
+    Python. (The maintainers currently use `pyenv` but Python's built-in
+    `venv` should work too.)
+
+1.  Run `pip install -r requirements.txt`
+
+1.  Run `./release/build_distribution.sh`
+
+1.  If the above succeeds, it will leave the wheel in `/tmp/tensorflow_quantum/`
+    on your system. Take note of the name of the wheel file that
+    `build_distribution.sh` prints when it finishes.
+
+1.  Run `./release/clean_distribution.sh /tmp/tensorflow_quantum/WHEEL_FILE`,
+    where `WHEEL_FILE` is the file noted in the previous step. If this works, it
+    will create a new wheel file in a subdirectory named `wheelhouse`. If an
+    error occurs, it will hopefully report the problem. If the error is a Python
+    platform tag mismatch, run `./release/clean_distribution.sh -s
+    /tmp/tensorflow_quantum/WHEEL_FILE`; this will run `auditwheel show` on the
+    wheel file to indicate what version of `manylinux` this wheel can be made to
+    run on if you use `auditwheel` to repair it. With that information, you may
+    be able to edit the `build_distribution.sh` script to experiment with
+    different values for the Crosstool and/or the Docker images used.
+
+1.  If the previous step succeeded, go to testing the release.
+
+1.  If the tests succeed, repeat the `build_distribution.sh` and
+    `clean_distribution.sh` steps for different versions of Python.

release/build_distribution.sh

@@ -0,0 +1,156 @@
+#!/bin/bash
+# Copyright 2025 The TensorFlow Quantum Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# =============================================================================
+
+# Summary: build a wheel for TFQ using a TensorFlow SIG Build container.
+# Run this script with the option "-h" to get a usage summary.
+#
+# To ensure binary compatibility with TensorFlow, TFQ distributions are built
+# using TensorFlow's SIG Build containers and Crosstool C++ toolchain. This
+# script encapsulates the process. The basic steps this script performs are:
+#
+# 1. Write to a file a small shell script that does the following:
+#
+#    a) pip install TFQ's requirements.txt file
+#    b) run TFQ's configure.sh script
+#    c) run Bazel to build build_pip_package
+#    d) run the resulting build_pip_package
+#    e) copy the wheel created by build_pip_package to ./wheels
+#
+# 2. Start Docker with image tensorflow/build:${tf_version}-python${py_version}
+#    and run the script written in step 1.
+#
+# 3. Do some basic tests on the wheel using standard Python utilities.
+#
+# 4. Exit.
+
+set -eu -o pipefail
+
+function quit() {
+  printf 'Error: %b\n' "$*" >&2
+  exit 1
+}
+
+# Find the top of the local TFQ git tree. Do it early in case this fails.
+thisdir=$(dirname "${BASH_SOURCE[0]:?}")
+repo_dir=$(git -C "${thisdir}" rev-parse --show-toplevel 2> /dev/null) || \
+  quit "This script must be run from inside the TFQ git tree."
+
+# Default values for variables that can be changed via command line flags.
+tf_version="2.16"
+py_version=$(python3 --version | cut -d' ' -f2 | cut -d. -f1,2)
+cuda_version="12"
+cleanup="true"
+
+usage="Usage: ${0} [OPTIONS]
+Build a distribution wheel for TensorFlow Quantum.
+
+Configuration options:
+  -c X.Y  Use CUDA version X.Y (default: ${cuda_version})
+  -p X.Y  Use Python version X.Y (default: ${py_version})
+  -t X.Y  Use TensorFlow version X.Y (default: ${tf_version})
+
+General options:
+  -e     Don't run bazel clean at the end (default: do)
+  -n     Dry run: print commands but don't execute them
+  -h     Show this help message and exit"
+
+dry_run="false"
+while getopts "c:ehnp:t:" opt; do
+  case "${opt}" in
+    c) cuda_version="${OPTARG}" ;;
+    e) cleanup="false" ;;
+    h) echo "${usage}"; exit 0 ;;
+    n) dry_run="true" ;;
+    p) py_version=$(echo "${OPTARG}" | cut -d. -f1,2) ;;
+    t) tf_version="${OPTARG}" ;;
+    *) quit "${usage}" ;;
+  esac
+done
+shift $((OPTIND -1))
+
+# See https://hub.docker.com/r/tensorflow/build/tags for available containers.
+docker_image="tensorflow/build:${tf_version}-python${py_version}"
+
+# This should match what TensorFlow's .bazelrc file uses.
+crosstool="@sigbuild-r${tf_version}-clang_config_cuda//crosstool:toolchain"
+
+# Note: configure.sh is run inside the container, and it creates a .bazelrc
+# file that adds other cxxopt flags. They don't need to be repeated here.
+BUILD_OPTIONS="--cxxopt=-O3 --cxxopt=-msse2 --cxxopt=-msse3 --cxxopt=-msse4"
+
+# Create a script to be run by the shell inside the Docker container.
+build_script=$(mktemp /tmp/tfq_build.XXXXXX)
+trap 'rm -f "${build_script}" || true' EXIT
+
+# The printf'ed section dividers are to make it easier to search the output.
+cat <<'EOF' > "${build_script}"
+#!/bin/bash
+set -o errexit
+cd /tfq
+PREFIX='[DOCKER] '
+exec > >(sed "s/^/${PREFIX} /")
+exec 2> >(sed "s/^/${PREFIX} /" >&2)
+printf ":::::::: Build configuration inside Docker container ::::::::\n"
+printf "  Docker image:     ${docker_image}\n"
+printf "  Crosstool:        ${crosstool}\n"
+printf "  TF version:       ${tf_version}\n"
+printf "  Python version:   ${py_version}\n"
+printf "  CUDA version:     ${cuda_version}\n"
+printf "  vCPUs available:  $(nproc)\n"
+printf "\n\n:::::::: Configuring Python environment ::::::::\n\n"
+python3 -m pip install --upgrade pip --root-user-action ignore
+pip install -r requirements.txt --root-user-action ignore
+printf "Y\n" | ./configure.sh
+printf "\n:::::::: Starting Bazel build ::::::::\n\n"
+bazel build ${build_flags} release:build_pip_package
+printf "\n:::::::: Creating Python wheel ::::::::\n\n"
+bazel-bin/release/build_pip_package /build_output/
+if [[ "${cleanup}" == "true" ]]; then
+  printf "\n:::::::: Cleaning up ::::::::\n\n"
+  bazel clean --async
+fi
+EOF
+
+chmod +x "${build_script}"
+
+# Use 'set --' to build the command in the positional parameters ($1, $2, ...)
+set -- docker run -it --rm --network host \
+  -w /tfq \
+  -v "${repo_dir}":/tfq \
+  -v /tmp/tensorflow_quantum:/build_output \
+  -v "${build_script}:/tmp/build_script.sh" \
+  -e HOST_PERMS="$(id -u):$(id -g)" \
+  -e build_flags="--crosstool_top=${crosstool} ${BUILD_OPTIONS}" \
+  -e cuda_version="${cuda_version}" \
+  -e py_version="${py_version}" \
+  -e tf_version="${tf_version}" \
+  -e docker_image="${docker_image}" \
+  -e crosstool="${crosstool}" \
+  -e cleanup="${cleanup}" \
+  "${docker_image}" \
+  /tmp/build_script.sh
+
+if [[ "${dry_run}" == "true" ]]; then
+  # Loop through the positional parameters and simply print them.
+  printf "(Dry run) "
+  printf '%s ' "$@"
+else
+  echo "Spinning up a Docker container with ${docker_image} …"
+  "$@"
+
+  echo "Done. Look for wheel in /tmp/tensorflow_quantum/."
+  ls -l /tmp/tensorflow_quantum/
+fi

release/build_pip_package.sh

@@ -17,21 +17,21 @@
 set -e
 
 # Pick the Python that TFQ/TensorFlow used during configure/build.
-# Order: explicit env -> python3 (>= 3.10)
+# Order: explicit env -> python3 (>= 3.9)
 PY="${PYTHON_BIN_PATH:-}"
 if [[ -z "${PY}" ]]; then
   if ! command -v python3 >/dev/null 2>&1; then
-    echo "ERROR: python3 not found. Set PYTHON_BIN_PATH to a Python 3.10+ interpreter." >&2
+    echo "ERROR: python3 not found. Set PYTHON_BIN_PATH to a Python 3.9+ interpreter." >&2
     exit 2
   fi
 
-  # Require Python >= 3.10 for TFQ.
+  # Require Python >= 3.9 for TFQ.
   if ! python3 - <<'PY'
 import sys
-sys.exit(0 if sys.version_info[:2] >= (3, 10) else 1)
+sys.exit(0 if sys.version_info[:2] >= (3, 9) else 1)
 PY
   then
-    echo "ERROR: Python 3.10+ required for TensorFlow Quantum; found $(python3 -V 2>&1)." >&2
+    echo "ERROR: Python 3.9+ required for TensorFlow Quantum; found $(python3 -V 2>&1)." >&2
     exit 2
   fi
 
@@ -40,7 +40,7 @@ fi
 echo "Using Python: ${PY}"
 
 # Ensure packaging tools are present in THIS interpreter.
-pip install -qq setuptools wheel build --root-user-action ignore
+"${PY}" -m pip install -qq setuptools wheel build --root-user-action ignore
 
 EXPORT_DIR="bazel-bin/release/build_pip_package.runfiles/__main__"
 
@@ -73,7 +73,7 @@ main() {
   cp dist/*.whl "${DEST}"
   popd
   rm -rf "${TMPDIR}"
-  echo $(date) : "=== Output wheel file is in: ${DEST}"
+  echo "$(date) : === Done."
 }
 
 main "$@"