Substantial update to the airbnb intro tutorial and related files

Author: mhauru

docs/source/index.rst

@@ -1,6 +1,6 @@
 .. _page-index:
 
-sqlsynthgen's Documentation
+sqlsynthgen
 ---------------------------
 
 **sqlsynthgen** is a package for making copies of relational databases and populating them with random data.

docs/source/installation.rst

@@ -3,9 +3,6 @@
 Installation
 ============
 
-End User
---------
-
 To use SqlSynthGen, first install it:
 
 .. code-block:: console

docs/source/introduction.rst

@@ -25,9 +25,9 @@ After :ref:`Installation <page-installation>`, we can run ``sqlsynthgen`` to see
      remove-vocab     Truncate all vocabulary tables in the dst schema.
      validate-config  Validate the format of a config file.
 
-For the simplest case, we will need `make-tables`, `make-generators`, `create-tables` and `create-data` but, first,
+For the simplest case, we will need ``make-tables``, ``make-generators``, ``create-tables`` and ``create-data`` but, first,
 we need to set environment variables to tell sqlsynthgen how to access our source database (where the real data resides now) and destination database (where the synthetic data will go).
-We can do that in the terminal with the `export` keyword, as shown below, or in a file called `.env`.
+We can do that in the terminal with the ``export`` keyword, as shown below, or in a file called ``.env``.
 The source and destination may be on the same database server, as long as the database or schema names differ.
 If the source and destination schemas are the default schema for the user on that database, you should not set those variables.
 If you are using a DBMS that does not support schemas (e.g. MariaDB), you must not set those variables.
@@ -40,39 +40,39 @@ If you are using a DBMS that does not support schemas (e.g. MariaDB), you must n
    $ export DST_DSN="postgresql://someuser:[email protected]/dst_db"
    $ export DST_SCHEMA='myschema'
 
-Next, we make a SQLAlchemy file that defines the structure of your database using the `make-tables` command:
+Next, we make a SQLAlchemy file that defines the structure of your database using the ``make-tables`` command:
 
 .. code-block:: console
 
    $ sqlsynthgen make-tables
 
-This will have created a file called `orm.py` in the current directory, with a SQLAlchemy class for each of your tables.
+This will have created a file called ``orm.py`` in the current directory, with a SQLAlchemy class for each of your tables.
 
 The next step is to make a sqlsynthgen file that defines one data generator per table in the source database:
 
 .. code-block:: console
 
    $ sqlsynthgen make-generators
 
-This will have created a file called `ssg.py` in the current directory.
+This will have created a file called ``ssg.py`` in the current directory.
 
-We can use the `create-table` command to read the `orm.py` file, create our destination schema (if it doesn't already exist) and to create empty copies of all the tables that in the source database.
+We can use the ``create-table`` command to read the ``orm.py`` file, create our destination schema (if it doesn't already exist), and to create empty copies of all the tables that are in the source database.
 
 .. code-block:: console
 
    $ sqlsynthgen create-tables
 
-Now that we have created the schema that will hold synthetic data, we can use the `create-data` command to read `orm.py` & `ssg.py` and generate data:
+Now that we have created the schema that will hold synthetic data, we can use the ``create-data`` command to read ``orm.py`` & ``ssg.py`` and generate data:
 
 .. code-block:: console
 
    $ sqlsynthgen create-data
 
-By default, `create-data` will have inserted one row per table and will have used the column data types to decide how to randomly generate data.
-To create more data each time we call `create-data`, we can provide an integer argument:
+By default, ``create-data`` will have inserted one row per table and will have used the column data types to decide how to randomly generate data.
+To create more data each time we call ``create-data``, we can provide the ``num-passes`` argument:
 
 .. code-block:: console
 
-   $ sqlsynthgen create-data 10
+   $ sqlsynthgen create-data --num-passes=10
 
 We will have inserted 11 rows per table, with the last two commands.

docs/source/quickstart.rst

@@ -1,22 +1,27 @@
-import random
 import datetime
+import random
 from typing import Optional, Generator, Tuple
 
+
 def user_dates_provider(generic):
     date_account_created: datetime.date = generic.datetime.date(start=2010, end=2015)
 
     booking_date: Optional[datetime.date] = None
     if generic.choice([True, False]):
         booking_date = generic.datetime.date(
             start=date_account_created.year + 1, end=2016
-         )
+        )
 
     return date_account_created, booking_date
 
+
 def user_age_provider(query_results):
-    mu: float = query_results[0][0]
-    sigma: float = query_results[0][1]
-    return random.gauss(mu, sigma)
+    # The [0] picks up the first row of the query results. This is needed because all
+    # query results are always tables, and could in principle have many rows.
+    mean: float = query_results[0]["mean"]
+    std_dev: float = query_results[0]["std_dev"]
+    return random.gauss(mean, std_dev)
+
 
 def sessions_story():
     """Generate users and their sessions."""
@@ -25,9 +30,7 @@ def sessions_story():
     # a new user will be sent back to us with our randomly chosen device type
     user: dict = yield (
         "users",  # table name
-        {
-            "first_device_type": random.choice(device_types)
-        }  # see 1. below
+        {"first_device_type": random.choice(device_types)},
     )
 
     # create between 10 and 19 sessions per user
@@ -39,15 +42,13 @@ def sessions_story():
             yield (
                 "sessions",
                 {
-                    "user_id": user["id"],  # see 2. below
+                    "user_id": user["id"],
                     "device_type": user["first_device_type"],
-                }
+                },
             )
         else:
             # ...but sometimes it is from any device type
             yield (
                 "sessions",
-                {
-                    "user_id": user["id"],
-                    "device_type": random.choice(device_types)},
+                {"user_id": user["id"], "device_type": random.choice(device_types)},
             )

tests/examples/airbnb/airbnb_generators.py

@@ -1,23 +1,9 @@
-
 row_generators_module: airbnb_generators
 
 tables:
   countries:
     vocabulary_table: true
 
-  sessions:
-    num_rows_per_pass: 0
-    row_generators:
-      - name: generic.numeric.integer_number
-        kwargs:
-          start: 0
-          end: 3600
-        columns_assigned: secs_elapsed
-      - name: generic.choice
-        kwargs:
-          items: ["show", "index", "personalize"]
-        columns_assigned: action
-
   age_gender_bkts:
     num_rows_per_pass: 1
     row_generators:
@@ -32,21 +18,29 @@ tables:
   users:
     num_rows_per_pass: 0
     row_generators:
-      - name: generic.person.password
-        kwargs: null
-        columns_assigned: id
-      - name: generic.person.identifier
+      - name: airbnb_generators.user_age_provider
         kwargs:
-           mask: '"@@##@@@@"' # Using this provider, we generate alpha-numeric IDs.
+          query_results: SRC_STATS["age_stats"]
+        columns_assigned: age
+      - name: generic.person.password
         columns_assigned: id
       - name: airbnb_generators.user_dates_provider
         kwargs:
            generic: generic
         columns_assigned: ["date_account_created", "date_first_booking"]
-      - name: airbnb_generators.user_age_provider
+
+  sessions:
+    num_rows_per_pass: 0
+    row_generators:
+      - name: generic.numeric.integer_number
         kwargs:
-           query_results: SRC_STATS["age_stats"]
-        columns_assigned: age
+          start: 0
+          end: 3600
+        columns_assigned: secs_elapsed
+      - name: generic.choice
+        kwargs:
+          items: ["show", "index", "personalize"]
+        columns_assigned: action
 
 src-stats:
   - name: age_stats
@@ -55,9 +49,9 @@ src-stats:
       FROM users
       WHERE age <= 100
     dp-query: >
-      SELECT AVG(age), STDDEV(age)
+      SELECT AVG(age) AS mean, STDDEV(age) AS std_dev
       FROM query_result
-    epsilon: 0.1
+    epsilon: 0.5
     delta: 0.000001
     snsql-metadata:
       max_ids: 1
@@ -70,6 +64,7 @@ src-stats:
         upper: 100
 
 story_generators_module: airbnb_generators
+
 story_generators:
   - name: airbnb_generators.sessions_story
     num_stories_per_pass: 30

tests/examples/airbnb/complete_config.yaml

@@ -111,12 +111,14 @@ def upload_csv_to_database(
     dataframe = dataframe.replace({np.nan: None})
 
     num_rows = len(dataframe)
-    for _, data_as_series in tqdm(dataframe.iterrows(), total=num_rows):
+    commit_frequency = 100000
+    for idx, row in tqdm(enumerate(dataframe.iterrows()), total=num_rows):
+        data_as_series = row[1]
         model_instance = mapped_class(**data_as_series)
         if filter_function(model_instance, session):
             session.add(model_instance)
-        else:
-            print(f"Skipping: {model_instance=}")
+        if idx % commit_frequency == 0:
+            session.commit()
     session.commit()
 
 

tests/examples/airbnb/csv_to_database.py

@@ -1,7 +1,7 @@
 """This file was auto-generated by sqlsynthgen but can be edited manually."""
 from mimesis import Generic
 from mimesis.locales import Locale
-from sqlsynthgen.base import FileUploader
+from sqlsynthgen.base import FileUploader, TableGenerator
 from sqlsynthgen.unique_generator import UniqueGenerator
 
 generic = Generic(locale=Locale.EN_GB)
@@ -12,6 +12,12 @@
 from sqlsynthgen.providers import ColumnValueProvider
 
 generic.add_provider(ColumnValueProvider)
+from sqlsynthgen.providers import NullProvider
+
+generic.add_provider(NullProvider)
+from sqlsynthgen.providers import SQLGroupByProvider
+
+generic.add_provider(SQLGroupByProvider)
 from sqlsynthgen.providers import TimedeltaProvider
 
 generic.add_provider(TimedeltaProvider)
@@ -34,7 +40,7 @@
 countries_vocab = FileUploader(orm.Countries.__table__)
 
 
-class age_gender_bktsGenerator:
+class age_gender_bktsGenerator(TableGenerator):
     num_rows_per_pass = 1
 
     def __init__(self):
@@ -52,23 +58,22 @@ def __call__(self, dst_db_conn):
         return result
 
 
-class usersGenerator:
+class usersGenerator(TableGenerator):
     num_rows_per_pass = 0
 
     def __init__(self):
         pass
 
     def __call__(self, dst_db_conn):
         result = {}
+        result["age"] = airbnb_generators.user_age_provider(
+            query_results=SRC_STATS["age_stats"]
+        )
         result["id"] = generic.person.password()
-        result["id"] = generic.person.identifier(mask="@@##@@@@")
         (
             result["date_account_created"],
             result["date_first_booking"],
         ) = airbnb_generators.user_dates_provider(generic=generic)
-        result["age"] = airbnb_generators.user_age_provider(
-            query_results=SRC_STATS["age_stats"]
-        )
         result["timestamp_first_active"] = generic.datetime.datetime()
         result["gender"] = generic.text.color()
         result["signup_method"] = generic.text.color()
@@ -86,7 +91,7 @@ def __call__(self, dst_db_conn):
         return result
 
 
-class sessionsGenerator:
+class sessionsGenerator(TableGenerator):
     num_rows_per_pass = 0
 
     def __init__(self):
@@ -123,7 +128,8 @@ def run_airbnb_generators_sessions_story(dst_db_conn):
 
 story_generator_list = [
     {
-        "name": run_airbnb_generators_sessions_story,
+        "function": run_airbnb_generators_sessions_story,
         "num_stories_per_pass": 30,
+        "name": "airbnb_generators.sessions_story",
     },
 ]