more flexible kinematic model + handling different regional decimal separators + Section on how to run faster inference in Readme

Author: davidpagnon

Pose2Sim/Demo_Batch/Config.toml

@@ -82,7 +82,7 @@ det_frequency = 4 # Run person detection only every N frames, and inbetween trac
 device = 'auto' # 'auto', 'CPU', 'CUDA', 'MPS', 'ROCM'
 backend = 'auto' # 'auto', 'openvino', 'onnxruntime', 'opencv'
 
-tracking_mode = 'sports2d' # 'sports2d' or 'deepsort'. 'deepsort' is slower but more robust in difficult configurations
+tracking_mode = 'sports2d' # 'none', 'sports2d' or 'deepsort'. 'deepsort' is slower but more robust in difficult configurations
 deepsort_params = """{'max_age':30, 'n_init':3, 'nms_max_overlap':0.8, 'max_cosine_distance':0.3, 'nn_budget':200, 'max_iou_distance':0.8}""" # """{dictionary between 3 double quotes}"""
                   # More robust in crowded scenes but Can be tricky to parametrize. More information there: https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/deepsort_tracker.py#L51
                   # Note: For faster and more robust tracking, use {'embedder_gpu': True, embedder':'torchreid'}, which uses the GPU and runs osnet_ain_x1_0 by default. requires `pip install torch torchvision torchreid gdown tensorboard`

Pose2Sim/Demo_Batch/Trial_1/Config.toml

@@ -81,7 +81,7 @@
 # device = 'auto' # 'auto', 'CPU', 'CUDA', 'MPS', 'ROCM'
 # backend = 'auto' # 'auto', 'openvino', 'onnxruntime', 'opencv'
 
-# tracking_mode = 'sports2d' # 'sports2d' or 'deepsort'. 'deepsort' is slower but more robust in difficult configurations
+# tracking_mode = 'sports2d' # 'none', 'sports2d' or 'deepsort'. 'deepsort' is slower but more robust in difficult configurations
 # deepsort_params = """{'max_age':30, 'n_init':3, 'nms_max_overlap':0.8, 'max_cosine_distance':0.3, 'nn_budget':200, 'max_iou_distance':0.8}""" # """{dictionary between 3 double quotes}"""
 #                   # More robust in crowded scenes but Can be tricky to parametrize. More information there: https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/deepsort_tracker.py#L51
 #                   # Note: For faster and more robust tracking, use {'embedder_gpu': True, embedder':'torchreid'}, which uses the GPU and runs osnet_ain_x1_0 by default. requires `pip install torch torchvision torchreid gdown tensorboard`

Pose2Sim/Demo_Batch/Trial_2/Config.toml

@@ -80,7 +80,7 @@ participant_mass = [70.0, 63.5]     # float (eg 70.0), or list of floats (eg [70
 # device = 'auto' # 'auto', 'CPU', 'CUDA', 'MPS', 'ROCM'
 # backend = 'auto' # 'auto', 'openvino', 'onnxruntime', 'opencv'
 
-# tracking_mode = 'sports2d' # 'sports2d' or 'deepsort'. 'deepsort' is slower but more robust in difficult configurations
+# tracking_mode = 'sports2d' # 'none', 'sports2d' or 'deepsort'. 'deepsort' is slower but more robust in difficult configurations
 # deepsort_params = """{'max_age':30, 'n_init':3, 'nms_max_overlap':0.8, 'max_cosine_distance':0.3, 'nn_budget':200, 'max_iou_distance':0.8}""" # """{dictionary between 3 double quotes}"""
 #                   # More robust in crowded scenes but Can be tricky to parametrize. More information there: https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/deepsort_tracker.py#L51
 #                   # Note: For faster and more robust tracking, use {'embedder_gpu': True, embedder':'torchreid'}, which uses the GPU and runs osnet_ain_x1_0 by default. requires `pip install torch torchvision torchreid gdown tensorboard`

Pose2Sim/Demo_MultiPerson/Config.toml

@@ -81,7 +81,7 @@ det_frequency = 4 # Run person detection only every N frames, and inbetween trac
 device = 'auto' # 'auto', 'CPU', 'CUDA', 'MPS', 'ROCM'
 backend = 'auto' # 'auto', 'openvino', 'onnxruntime', 'opencv'
 
-tracking_mode = 'sports2d' # 'sports2d' or 'deepsort'. 'deepsort' is slower but more robust in difficult configurations
+tracking_mode = 'sports2d' # 'none', 'sports2d' or 'deepsort'. 'deepsort' is slower but more robust in difficult configurations
 deepsort_params = """{'max_age':30, 'n_init':3, 'nms_max_overlap':0.8, 'max_cosine_distance':0.3, 'nn_budget':200, 'max_iou_distance':0.8}""" # """{dictionary between 3 double quotes}"""
                   # More robust in crowded scenes but Can be tricky to parametrize. More information there: https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/deepsort_tracker.py#L51
                   # Note: For faster and more robust tracking, use {'embedder_gpu': True, embedder':'torchreid'}, which uses the GPU and runs osnet_ain_x1_0 by default. requires `pip install torch torchvision torchreid gdown tensorboard`

Pose2Sim/Demo_SinglePerson/Config.toml

@@ -82,7 +82,7 @@ det_frequency = 4 # Run person detection only every N frames, and inbetween trac
 device = 'auto' # 'auto', 'CPU', 'CUDA', 'MPS', 'ROCM'
 backend = 'auto' # 'auto', 'openvino', 'onnxruntime', 'opencv'
 
-tracking_mode = 'sports2d' # 'sports2d' or 'deepsort'. 'deepsort' is slower but more robust in difficult configurations
+tracking_mode = 'sports2d' # 'none', 'sports2d' or 'deepsort'. 'deepsort' is slower but more robust in difficult configurations
 deepsort_params = """{'max_age':30, 'n_init':3, 'nms_max_overlap':0.8, 'max_cosine_distance':0.3, 'nn_budget':200, 'max_iou_distance':0.8}""" # """{dictionary between 3 double quotes}"""
                   # More robust in crowded scenes but Can be tricky to parametrize. More information there: https://github.com/levan92/deep_sort_realtime/blob/master/deep_sort_realtime/deepsort_tracker.py#L51
                   # Note: For faster and more robust tracking, use {'embedder_gpu': True, embedder':'torchreid'}, which uses the GPU and runs osnet_ain_x1_0 by default. requires `pip install torch torchvision torchreid gdown tensorboard`

Pose2Sim/kinematics.py

@@ -47,6 +47,9 @@
                             best_coords_for_measurements, compute_height
 from Pose2Sim.skeletons import *
 
+import locale 
+locale.setlocale(locale.LC_NUMERIC, 'C')
+
 
 ## AUTHORSHIP INFORMATION
 __author__ = "Ivan Sun, David Pagnon"
@@ -110,30 +113,21 @@ def get_markers_path(pose_model, osim_setup_dir):
     - markers_path: (Path) Path to the marker file.
     '''
 
-    if pose_model == 'BODY_25B':
-        marker_file = 'Markers_Body25b.xml'
-    elif pose_model == 'BODY_25':
-        marker_file = 'Markers_Body25.xml'
-    elif pose_model == 'BODY_135':
-        marker_file = 'Markers_Body135.xml'
-    elif pose_model == 'BLAZEPOSE':
-        marker_file = 'Markers_Blazepose.xml'
-    elif pose_model == 'HALPE_26':
-        marker_file = 'Markers_Halpe26.xml'
-    elif pose_model == 'HALPE_68' or pose_model == 'HALPE_136':
-        marker_file = 'Markers_Halpe68_136.xml'
-    elif pose_model == 'COCO_133' or pose_model == 'COCO_133_WRIST':
-        marker_file = 'Markers_Coco133.xml'
-    # elif pose_model == 'COCO' or pose_model == 'MPII':
-    #     marker_file = 'Markers_Coco.xml'
-    elif pose_model == 'COCO_17':
-        marker_file = 'Markers_Coco17.xml'
-    elif pose_model == 'LSTM':
-        marker_file = 'Markers_LSTM.xml'
+    pose_model = ''.join(pose_model.split('_')).lower()
+    if pose_model == 'halpe68' or pose_model == 'halpe136':
+        marker_file = 'Markers_Halpe68_136.xml'.lower()
+    elif pose_model == 'coco133' or pose_model == 'coco133wrist':
+        marker_file = 'Markers_Coco133.xml'.lower()
     else:
-        raise ValueError(f"Pose model '{pose_model}' not supported yet.")
+        marker_file = f'Markers_{pose_model}.xml'.lower()
 
-    markers_path = osim_setup_dir / marker_file
+    try:
+        markers_path = [
+            f for f in osim_setup_dir.glob('Markers_*.xml')
+            if f.name.lower() == marker_file
+        ][0]
+    except:
+        raise ValueError(f"Pose model '{pose_model}' not supported yet.")
 
     return markers_path
 
@@ -150,30 +144,21 @@ def get_scaling_setup(pose_model, osim_setup_dir):
     - scaling_setup_path: (Path) Path to the OpenSim scaling setup file.
     '''
 
-    if pose_model == 'BODY_25B':
-        scaling_setup_file = 'Scaling_Setup_Pose2Sim_Body25b.xml'
-    elif pose_model == 'BODY_25':
-        scaling_setup_file = 'Scaling_Setup_Pose2Sim_Body25.xml'
-    elif pose_model == 'BODY_135':
-        scaling_setup_file = 'Scaling_Setup_Pose2Sim_Body135.xml'
-    elif pose_model == 'BLAZEPOSE':
-        scaling_setup_file = 'Scaling_Setup_Pose2Sim_Blazepose.xml'
-    elif pose_model == 'HALPE_26':
-        scaling_setup_file = 'Scaling_Setup_Pose2Sim_Halpe26.xml'
-    elif pose_model == 'HALPE_68' or pose_model == 'HALPE_136':
-        scaling_setup_file = 'Scaling_Setup_Pose2Sim_Halpe68_136.xml'
-    elif pose_model == 'COCO_133' or pose_model == 'COCO_133_WRIST':
-        scaling_setup_file = 'Scaling_Setup_Pose2Sim_Coco133.xml'
-    # elif pose_model == 'COCO' or pose_model == 'MPII':
-    #     scaling_setup_file = 'Scaling_Setup_Pose2Sim_Coco.xml'
-    elif pose_model == 'COCO_17':
-        scaling_setup_file = 'Scaling_Setup_Pose2Sim_Coco17.xml'
-    elif pose_model == 'LSTM':
-        scaling_setup_file = 'Scaling_Setup_Pose2Sim_LSTM.xml'
+    pose_model = ''.join(pose_model.split('_')).lower()
+    if pose_model == 'halpe68' or pose_model == 'halpe136':
+        scaling_setup_file = 'Scaling_Setup_Pose2Sim_Halpe68_136.xml'.lower()
+    elif pose_model == 'coco133' or pose_model == 'coco133wrist':
+        scaling_setup_file = 'Scaling_Setup_Pose2Sim_Coco133.xml'.lower()
     else:
-        raise ValueError(f"Pose model '{pose_model}' not supported yet.")
+        scaling_setup_file = f'Scaling_Setup_Pose2Sim_{pose_model}.xml'.lower()
 
-    scaling_setup_path = osim_setup_dir / scaling_setup_file
+    try:
+        scaling_setup_path = [
+            f for f in osim_setup_dir.glob('Scaling_Setup_Pose2Sim_*.xml')
+            if f.name.lower() == scaling_setup_file
+        ][0]
+    except:
+        raise ValueError(f"Pose model '{pose_model}' not supported yet.")
 
     return scaling_setup_path
 
@@ -190,30 +175,24 @@ def get_IK_Setup(pose_model, osim_setup_dir):
     - ik_setup_path: (Path) Path to the OpenSim IK setup file.
     '''
 
-    if pose_model == 'BODY_25B':
-        ik_setup_file = 'IK_Setup_Pose2Sim_Body25b.xml'
-    elif pose_model == 'BODY_25':
-        ik_setup_file = 'IK_Setup_Pose2Sim_Body25.xml'
-    elif pose_model == 'BODY_135':
-        ik_setup_file = 'IK_Setup_Pose2Sim_Body135.xml'
-    elif pose_model == 'BLAZEPOSE':
-        ik_setup_file = 'IK_Setup_Pose2Sim_Blazepose.xml'
-    elif pose_model == 'HALPE_26':
-        ik_setup_file = 'IK_Setup_Pose2Sim_Halpe26.xml'
-    elif pose_model == 'HALPE_68' or pose_model == 'HALPE_136':
-        ik_setup_file = 'IK_Setup_Pose2Sim_Halpe68_136.xml'
-    elif pose_model == 'COCO_133' or pose_model == 'COCO_133_WRIST':
-        ik_setup_file = 'IK_Setup_Pose2Sim_Coco133.xml'
-    # elif pose_model == 'COCO' or pose_model == 'MPII':
-    #     ik_setup_file = 'IK_Setup_Pose2Sim_Coco.xml'
-    elif pose_model == 'COCO_17':
-        ik_setup_file = 'IK_Setup_Pose2Sim_Coco17.xml'
-    elif pose_model == 'LSTM':
-        ik_setup_file = 'IK_Setup_Pose2Sim_withHands_LSTM.xml'
+    pose_model = ''.join(pose_model.split('_')).lower()
+    if pose_model == 'halpe68' or pose_model == 'halpe136':
+        ik_setup_file = 'IK_Setup_Pose2Sim_Halpe68_136.xml'.lower()
+    elif pose_model == 'coco133' or pose_model == 'coco133wrist':
+        ik_setup_file = 'IK_Setup_Pose2Sim_Coco133.xml'.lower()
+    elif pose_model == 'lstm':
+        ik_setup_file = 'IK_Setup_Pose2Sim_withHands_LSTM.xml'.lower()
     else:
+        ik_setup_file = f'IK_Setup_Pose2Sim_{pose_model}.xml'.lower()
+
+    try:
+        ik_setup_path = [
+            f for f in osim_setup_dir.glob('IK_Setup_Pose2Sim_*.xml')
+            if f.name.lower() == ik_setup_file
+        ][0]
+    except:
         raise ValueError(f"Pose model '{pose_model}' not supported yet.")
 
-    ik_setup_path = osim_setup_dir / ik_setup_file
     return ik_setup_path
 
 
@@ -681,6 +660,10 @@ def kinematics_all(config_dict):
         logging.info(f"\tScaled model saved to {(kinematics_dir / (trc_file.stem + '_scaled.osim')).resolve()}")
 
         logging.info("\nInverse Kinematics...")
+        import time
+        start_time = time.time()
         perform_IK(trc_file, kinematics_dir, osim_setup_dir, pose_model, remove_IK_setup=remove_IK_setup)
+        end_time = time.time()
+        print(f"\tIK took {round(end_time - start_time, 2)} seconds for {trc_file.name}.")
         logging.info(f"\tDone. OpenSim logs saved to {opensim_logs_file.resolve()}.")
         logging.info(f"\tJoint angle data saved to {(kinematics_dir / (trc_file.stem + '.mot')).resolve()}\n")

Pose2Sim/poseEstimation.py

@@ -258,6 +258,8 @@ def process_video(video_path, pose_tracker, pose_model, output_format, save_vide
                 if tracking_mode == 'sports2d': 
                     if 'prev_keypoints' not in locals(): prev_keypoints = keypoints
                     prev_keypoints, keypoints, scores = sort_people_sports2d(prev_keypoints, keypoints, scores=scores)
+                else:
+                    pass
 
                 # Save to json
                 if 'openpose' in output_format:

README.md

@@ -72,6 +72,7 @@ Pose2Sim stands for "OpenPose to OpenSim", as it originally used *OpenPose* inpu
    3. [Demonstration Part-2: Visualize your results with OpenSim or Blender](#demonstration-part-2-visualize-your-results-with-opensim-or-blender)
    4. [Demonstration Part-3: Try multi-person analysis](#demonstration-part-3-try-multi-person-analysis)
    5. [Demonstration Part-4: Try batch processing](#demonstration-part-4-try-batch-processing)
+   6. [Too slow for you?](#too-slow-for-you)
 2. [Use on your own data](#use-on-your-own-data)
    1. [Setting up your project](#setting-up-your-project)
    2. [2D pose estimation](#2d-pose-estimation)
@@ -178,7 +179,7 @@ Type `ipython`, and try the following code:
 from Pose2Sim import Pose2Sim
 Pose2Sim.calibration()
 Pose2Sim.poseEstimation()
-Pose2Sim.synchronization() # When the GUI is prompted, make sure only RWrist point is selected
+Pose2Sim.synchronization()
 Pose2Sim.personAssociation()
 Pose2Sim.triangulation()
 Pose2Sim.filtering()
@@ -315,6 +316,44 @@ Run Pose2Sim from the `BatchSession` folder if you want to batch process the who
 For example, try uncommenting `[project]` and set `frame_range = [10,99]`, or uncomment `[pose]` and set `mode = 'lightweight'` in the `Config.toml` file of `Trial_2`.
 
 
+<br/>
+
+## Too slow for you?
+
+**Quick fixes:**
+- `Pose2Sim.calibration()`:\
+  Run it only when your cameras are moved or changed. If they are not, just copy a previous calibration.toml file into your new calibration folder.
+- `Pose2Sim.poseEstimation()`:
+  - Set `det_frequency = 100` in Config.toml. Do not run the person detector every frame. Run it once, and then track the bounding boxes, which is much faster (pose estimation will still be run every frame): .\
+  *150 s -> 30 s on my laptop with the Demo videos*
+  - Use `mode = 'lightweight'`: Will use a lighter version of RTMPose, which is faster but less accurate\
+  *30 s -> 20 s*
+  - Set `display_detection = false`. Do not display results in real time\
+  *20 s -> 15 s*
+  - Set `save_video = 'none'`. Do not save images and videos\
+  *15 s -> 9 s*
+  - Set `tracking_mode = 'sports2d'` or `tracking_mode = 'none'`. If several persons are in the scene, use the sports2d tracker or no tracker at all, but not 'deepsort' (sports2d tracking is almost instantaneous though).
+- `Pose2Sim.synchronization()`:\
+  Do not run if your cameras are natively synchronized.
+- `Pose2Sim.personAssociation()`:\
+  Do not run if there is only one person in the scene.
+- `Pose2Sim.triangulation()`:\
+  Not much to do here.
+- `Pose2Sim.filtering()`:\
+  You can skip this step, but it is quite fast already.
+- `Pose2Sim.markerAugmentation()`:\
+  Very fast, too. Note that marker augmentation won't necessarily improve results so you can consider skipping it.
+- `Pose2Sim.kinematics()`:\
+  Set `use_simple_model = true`. Use a simpler OpenSim model, without muscles and constraints. Note that the spine will be stiff and shoulders will be a simple ball joint, but this is accurate enough for most gait-related tasks\
+  9 s -> 0.7 s
+
+<br> 
+
+**Use your GPU**:\
+This would make pose estimation significantly faster, without any impact on accuracy. See [Installation](#installation) section for more information.
+
+
+
 </br></br>
 
 # Use on your own data