Move tests/examples to examples. Add READMEs to all examples.

Author: mhauru

.pre-commit-config.yaml

@@ -49,28 +49,34 @@ repos:
         types: ['python']
         exclude: (?x)(
           tests/examples|
-          tests/workspace
+          tests/workspace|
+          examples
           )
       - id: pylint
         name: Pylint
         entry: poetry run pylint
         language: system
         types: ['python']
+        exclude: (?x)(
+          examples/
+          )
       - id: pydocstyle
         name: pydocstyle
         entry: poetry run pydocstyle
         language: system
         types: ['python']
         exclude: (?x)(
           docs/|
-          tests/
+          tests/|
+          examples/
           )
       - id: mypy
         name: mypy
         entry: poetry run mypy --follow-imports=silent
         language: system
         exclude: (?x)(
           tests/examples|
-          tests/workspace
+          tests/workspace|
+          examples
           )
         types: ['python']

docs/source/introduction.rst

@@ -11,7 +11,7 @@ This is done in a manner that maintains transparency and control over how the or
 
 In this tutorial, we go through the different mechanisms SSG has for configuring the data generation, and the different levels of fidelity they can provide and different kinds of utility they can have.
 To showcase SSG, we will use the `AirBnb User Bookings dataset, available at Kaggle <https://www.kaggle.com/competitions/airbnb-recruiting-new-user-bookings/data>`_.
-The original dataset is a collection CSV files that can be ported to a relational database using `this Python script <https://github.com/alan-turing-institute/sqlsynthgen/blob/migrate-adult-dataset-to-SQL/tests/examples/airbnb/csv_to_database.py>`_ (it requires having SSG `previously installed <https://sqlsynthgen.readthedocs.io/en/latest/installation.html#enduser>`_).
+The original dataset is a collection CSV files that can be ported to a relational database using `this Python script <https://github.com/alan-turing-institute/sqlsynthgen/blob/main/examples/airbnb/csv_to_database.py>`_ (it requires having SSG `previously installed <https://sqlsynthgen.readthedocs.io/en/latest/installation.html#enduser>`_).
 The script assumes you have a local PostgresSQL server running at port 5432, username ``postgres`` and password ``password``, with a database called ``airbnb`` to upload the data to.
 These assumptions can be edited in the ``main`` function of the script.
 
@@ -88,7 +88,7 @@ The ``generic`` object on line 9 is an instance of the Mimesis type `generic pro
 Mimesis is a package for creating random data and has a wide array of providers (the Mimesis term for data generators) for different scenarios, which SSG makes extensive use of.
 
 Similar edits as above for the ``users`` table need to be made for the primary key columns of the other tables.
-See `this Python file <https://github.com/alan-turing-institute/sqlsynthgen/blob/migrate-adult-dataset-to-SQL/tests/examples/airbnb/ssg_manual_edit.py>`_ for the full changes to the ``ssg.py`` file.
+See `this Python file <https://github.com/alan-turing-institute/sqlsynthgen/blob/main/examples/airbnb/ssg_manual_edit.py>`_ for the full changes to the ``ssg.py`` file.
 
 Now when we run ``create-data`` we get valid, if not very sensible, values in each of our tables. For example:
 
@@ -585,4 +585,4 @@ Note that we make here the same trade off as we did before: generating very high
 * Full transparency and control over the ways in which the source data is utilised, and thus the ways in which privacy could in principle be at risk, including easy implementation of differential privacy guarantees.
 * The possibility of starting from very low fidelity data, and incrementally adding fidelity to particular aspects of the data, as is needed to serve the utility of whatever use case the synthetic data is created for.
 
-Examples of the complete files generated by the tutorial can be found at: ``/sqlsynthgen/tests/examples/airbnb``.
+Examples of the complete files generated by the tutorial can be found at: ``/sqlsynthgen/examples/airbnb``.

docs/source/loan_data.rst

@@ -68,7 +68,7 @@ we see that they are always 0 or 1 so we will pick randomly from 0 and 1 for our
 
 **config.yaml**
 
-.. literalinclude:: ../../tests/examples/loans/config1.yaml
+.. literalinclude:: ../../examples/loans/config1.yaml
    :language: yaml
 
 We run SqlSynthGen's ``make-generators`` command to create ``ssg.py``, which contains a generator class for each table in the source database:
@@ -101,7 +101,7 @@ We notice that the ``districts`` table doesn't contain any sensitive data so we
 
 **config.yaml**
 
-.. literalinclude:: ../../tests/examples/loans/config2.yaml
+.. literalinclude:: ../../examples/loans/config2.yaml
    :language: yaml
 
 We can export the vocabularies to `.yaml` files, delete the old synthetic data, import the vocabularies and create new synthetic data with:
@@ -202,14 +202,14 @@ We can take the real values in the right proportions, and even add noise to make
 
 **config.yaml**
 
-.. literalinclude:: ../../tests/examples/loans/config3.yaml
+.. literalinclude:: ../../examples/loans/config3.yaml
    :language: yaml
 
 We define a custom row-generator to use the source statistics and Python's ``random.choices()`` function to choose a value:
 
 **my_row_generators.py**
 
-.. literalinclude:: ../../tests/examples/loans/my_row_generators.py
+.. literalinclude:: ../../examples/loans/my_row_generators.py
    :language: python
 
 As before, we will need to re-create ``ssg.py`` and the data.

examples/README.md

@@ -0,0 +1,3 @@
+# Examples
+
+This folder holds example configurations of how sqlsynthgen may be used on various data schemas.

examples/airbnb/README.md

@@ -0,0 +1,3 @@
+# Introductory example config using AirBnB data
+
+This is the SSG configuration developed in the [introductory tutorial](https://sqlsynthgen.readthedocs.io/en/latest/introduction.html) in our docs.

…sts/examples/airbnb/airbnb_generators.py examples/airbnb/airbnb_generators.pytests/examples/airbnb/airbnb_generators.py renamed to examples/airbnb/airbnb_generators.py tests/examples/airbnb/airbnb_generators.py renamed to examples/airbnb/airbnb_generators.py

@@ -1,6 +1,6 @@
 import datetime
 import random
-from typing import Optional, Generator, Tuple
+from typing import Generator, Optional, Tuple
 
 
 def user_dates_provider(generic):

…sts/examples/airbnb/complete_config.yaml examples/airbnb/complete_config.yamltests/examples/airbnb/complete_config.yaml renamed to examples/airbnb/complete_config.yaml tests/examples/airbnb/complete_config.yaml renamed to examples/airbnb/complete_config.yaml

@@ -1,7 +1,7 @@
 from typing import Any, Callable, List, Type
+
 import numpy as np
 import pandas as pd
-from tqdm import tqdm
 from sqlalchemy import (
     Column,
     Date,
@@ -14,6 +14,7 @@
 )
 from sqlalchemy.ext.declarative import declarative_base
 from sqlalchemy.orm import Session
+from tqdm import tqdm
 
 Base = declarative_base()