3DTracker-FAB documentation

The document was written by Dr. Jumpei Matsumoto and Dr. Marcelo Aguilar-Rivera. We thank Dr. Laleh Quinn for her proof reading.

Introduction

The 3DTracker-FAB (for animal behavior) is an open source software for 3D-video based computerized behavioral analysis. The 3DTracker-FAB will help users acquire and analyze 3D trajectories of animal body parts, as shown below.

_images/fig0-1.png

[Fig 0-1]

First, animals 3D imagines are reconstructed from depth cameras (~4) images (Fig 0, left). Second, body positions are estimated by fitting skeletal models to the 3D image (Fig 0, middle). Third, several behavioral parameters are calculated based on body part trajectories (Fig 0, right).

In the current workflow, a set of software modules are used. The images from multiple cameras are calibrated and recorded, with the “Recorder” module (Chapter 3). Then, the recorded images from different cameras are processed off-line for their integration into a full 3d image, down sampling, etc., with the “Preprocessor” module (Chapter 4). Then, the skeletal models are fitted to the 3D image, with the “Tracker” module (Chapter 5). Finally, the resultant 3D trajectories (csv file format) is analyzed with Python/MATLAB scripts (Chapter 6). The present version of the 3DTracker-FAB only supports the analysis of rats and mice. The scripts for post-processing of the trajectories are in preparation. Other species could be supported in the future.

The 3DTracker-FAB is basically licensed under an MIT license. So, note that the software is provided “as is”, without warranty of any kind, express or implied.

Index

Hardware preparation

Cameras

We suggest using Intel realsense R200 cameras (Fig 1-1) because of their outstanding performance characterized by the lack of interference among cameras when multiple units are simultaneously capturing the same object (see Appendix for details of the camera). A R200 is composed of a color camera and a depth sensor to obtain RGB images and depth images, respectively.

_images/fig1-1.png

[Fig 1-1: Intel realsense R200 camera]

# Unfortunately, R200 is now discontinued. However, there are still some stock in the intel store as part of a kit (https://click.intel.com/intelr-realsensetm-robotic-development-kit-2351.html) for robotics. In addition, sometimes it is possible to find them at Mouser Electronics. Given that the original USB 3.0 cable of the R200is too short, it is necessary to extend it. We suggest using an “active” repeater cable as found in the link (https://www.amazon.com/dp/B01FQ88CE6)

PC

The 3DTracker (except for MATLAB/Python scripts) runs only on Windows 10 (64-bit). The CPU does not need to be very fast. However, the number of cores (number of parallel processers?) in the CPU should be => than the number of cameras used, because the Recorder software processes each camera data in parallel. In addition, a large HDD should be installed in the PC, because of the big size of 3D video. The PC monitor resolution should be more than 1280x1024. We have been recording from four R200 cameras with a HP EliteDesk 800 G2 Core i7-6700 (3.4 GHz), 8GB RAM, 500 GB + 6TB HDD, and a power unit of 400W 80PLUS.

R200 uses almost the whole transmission bandwidth of an USB 3.0 port. So, more than one R200 cannot be connected to the same USB controller. For this reason, we added three expansion cards (https://www.amazon.co.jp/dp/B00B6ZCNGM) to get in total four USB controllers (including the one embedded in the mother board), thus connecting one R200 per controller. The expansion cards have to be hooked to the PC power supply to feed enough power into the “active” USB extension cable, beside they are inserted in the mother board slots.

Note

We strongly recommend using the same components suggested above to get a high-quality signal. We have tried several different products and those shown above are the best ones based on our experience.

Camera mounts

_images/fig1-2.png

[Fig 1-2]

Example of four depth cameras setup. Each camera was hooked to a post fixed to the floor and the ceiling (insert orange frame), through a custom adaptor made by gluing a 1/4-inch nut to a metal bulldog clip (insert blue frame). The main picture shows two of the four cameras.

Hint

The depth sensor of the camera measures the depth by emitting infrared (IR) light and capturing the IR reflection to objects (see Appendix for the detail of the mechanism of the camera). So, two cameras should not be placed face-to-face, to prevent the infrared sensor from saturating because of directly received infrared light from the other. Similarly, shiny surfaces should not be placed in front of a camera. Thus, we recommend placing the cameras slightly tilted downward from the horizontal plane. The depth sensor of R200 can’t detect a depth < 40 cm. The horizontal and vertical field of view of R200 is 59 ± 5 and 46 ± 5 degrees, respectively. These constraints should be considered during camera installation.

Software preparation

Installing 3rd party libraries

The 3DTracker-FAB utilizes functions of many free software libraries. Please download and install the software below. You can download the software from our website at http://3dtracker.org/downloads.html

  1. OpenCV 3.2.0 (library for image processing)
  2. PCL 1.8.0 (library for point cloud processing)
  3. Visual C++ 2015 redistributable (library for running the software that was written on Visual Studio C++ 2015)
  4. Kinect for Windows Runtime v1.8 (library for Kinect v1 camera and backward compatibility)

Warning

DO NOT install the R200 driver provided by Intel because the 3DTracker doesn’t need it. Also, some users have reported that updating the R200 firmware could trigger the malfunctioning of the R200 (see https://github.com/IntelRealSense/librealsense/issues/163; https://github.com/IntelRealSense/librealsense/issues/429).

After installing the libraries above, add the following folder to the environment variable “PATH”:

  • C:\opencv320\build\x64\vc14\bin (change “C:\opencv320”, to your install directory)
  • C:\Program Files\PCL 1.8.0\bin
  • C:\Program Files\OpenNI2\Redist

See how to add directories to the “PATH”: https://www.computerhope.com/issues/ch000549.htm

Installing main programs of the 3DTracker-FAB

Download the prebuilt version of the 3DTracker-FAB from the website http://3dtracker.org/downloads.html Uncompress the downloaded zip file. The extracted folder contains the software and data folder.

Hint

Because recorded data will be initially saved in the data folder, it is recommended to place the extracted data folder in a large HDD.

Recorder usage

The Recorder is a module for calibration of the multiple-cameras setup of the 3DTracker system.

Starting the Recorder

Connect the cameras to the PC and start Recorder.exe. If Recorder.exe doesn’t start, the cameras may have not been recognized (see Chapter 7 for the trouble shooting). Once the Recorder starts correctly, the following screen will appear:

_images/fig3-1.png

[Fig 3-1]

You will see a menu-bar, and a 3D view display where camera images are projected as a cloud of points. The 3D view can be controlled with the right button (R) dragging of a mouse, as follows: R + drag -> Rotate the view; R + drag + Ctrl key -> Zoom in and out; R + drag + Shift key -> translate the view. The red, green and blue axes shown in the 3D view represent X, Y, and Z axes, respectively. The remaining tool windows can be displayed from the menu tab > ‘Window’.

Camera calibration and configuration

If you look at the point cloud just after starting the Recorder, you will see that the points from different cameras are not correctly aligned. To discriminate clearly between the points from different cameras, you can color the points by enabling “Mark cameras with different colors” in the “Point cloud appearance” window. The misalignment is because of the lack of information about the relative positions of the cameras. The Recorder can estimate the relative positions through a semi-automatic calibration, as follows. During calibration the user scans the recording space with a light pointer, and records the pointer track on the “Calibration window”. Then, the recorder calculates the relative camera positions fitting the observed trajectories recorded from the different cameras during the calibration process. The pointer can be made by epoxying a white ping-pong ball of 40 mm of diameter (Fig 3-2), onto the tip of a white LED flashlight (https://www.amazon.com/dp/B00BIG5JQK/). The ping-pong ball works as a white light diffuser and an IR reflector.

_images/fig3-2.png

[Fig 3-2: left, light pointer; right, schematic showing the mechanism. The lines with different colors represent the pointer trajectories observed simultaneously from 3 different cameras in this example.]

The Recorder estimates the center of the pointer based on the closest blight point from a camera and a radius of the ping-pong ball, as shown below.

_images/fig3-3.png

[Fig 3-3]

Display the “Calibration window” from the “Window” menu. “Color filtering” setting section is used for detecting bright points that belong to the pointer. Find the best brightness threshold (the value of “Min Brightness” slider) by enabling “Test” check box.

The “Outlier removal” setting section below the “Color filtering” section is used to remove minor noise referred here as small clusters of blight points because of light reflections, etc. For details about the parameters of the Outlier removal, check here (http://pointclouds.org/documentation/tutorials/statistical_outlier.php). Usually, changing only the “Threshold” parameter is fine. You can also check the effect by enabling the check box “Test”.

After adjusting filter settings for the pointer detection, click the “Start pointer tracking” button on the bottom in the “Calibration” window. Then slowly move the pointer in the recording space to draw a trajectory for calibration (a spiral trajectory is recommended). Note that the pointer should be in view of all the cameras. To check the view from each camera, display “Camera Monitor” window.

After drawing the trajectory, click “Stop pointer tracking” button, and filter the trajectories to remove noise in the trajectories in the ‘following window’. Then, click “Use the traces for calibration” to calculate the relative camera positions. If this operation has been successfully done, the point clouds from different cameras should be perfectly aligned. To check the alignment, place some objects in front of the camera and color the points by enabling “Mark cameras with different colors” in the “Point cloud appearance” window. Then, look at the corners and the overall shapes of the objects in the screen. If point clouds with different colors are not well aligned, repeat the operation described above until a good result is achieved. The calibration using the pointer is only for the relative positions among cameras. So, the absolute position of one of the cameras (the reference camera) should be input manually. Move the sliders in the “Reference camera position & pose” section on the top of the “Calibration” window. Set X axis rotation, Y axis rotation, Z axis rotation, and translation. Rotation after translation makes it hard to adjust the position and rotation (see also the video link above). When you set the absolute position, make sure that the Y axis corresponds to the vertical axis, because the Tracker assumes it for fitting purposes. Lastly, set the region of interest (ROI) with the sliders on the top of the “Recording” window. You can also input the value directly by clicking + Ctrl key on slider. The same setting is expected to be useful for the coming recording sessions run with the same setup (unless the camera has been moved). To save and load the setting, use “File menu > save config” and “File menu > load config”, respectively. See Chapter 7 for trouble shooting calibration. For more information about calibration see the video at https://youtu.be/04JgLTaYI2c

Recording

To start recording, in the “Recording window”, input the data file name in the “session name” text box and click “Start recording RGBD” or “Start Recording Point cloud” button. With “Start recording RGBD”, raw data consisting of RGB images and depth (D) images is stored. The size of the raw data is relatively big (around 300 MB/camera/min). With “Start Recording Point cloud”, the point cloud data captured from the cameras are stored. Combined with the “ROI filtering” which removes the points outside the ROI, and “Voxel grid filtering” which down samples the points, the size of the point cloud data becomes much smaller than that of the raw RGBD data. Use 5 and 10 mm for the voxel size for the voxel grid filtering for mice and rats, respectively. By enabling the “save 2D videos” check boxes for the cameras, you can also save the 2D RGB video, which is useful for tracking (see Chapter 5), preliminary visual observation of behavior, etc. If you choose recording the raw data (RGBD), you can generate the 2D video offline. So, in this case, it is recommended not to check those boxes to reduce the computational load during recording.

Warning

if you work with the session name that was used previously, the original data will be overwritten.

After starting recording, 3D view is disabled and an RGB image is shown instead, reducing the computational load during recording. Click “Stop” button to stop recording. The recorded data is stored in the “data” folder containing the Recorder app (“Recorder.exe”) in the folder named as the “session name” that you have input before recording. See Chapter 9 for details about the output files.

Preprocessor usage

This module is for preprocessing the recorded data, for merging the 3D point clouds, down sampling (removing overlapping points) the data, calculating the surface normal vector at each point, etc. The preprocessing is necessary for pose estimation in the Tracker (Chapter 5). When you start the Preprocessor, it asks to select a folder to processes the session data in. If you select a folder containing multiple sub-folders from different sessions, the module will processes the data in these sub-folders if they don’t contain additional sub-levels. This is because the app does not search more than one level below the folder selected. The data preprocessed previously is skipped by default. The Preprocessor is called from the command prompt, and the preprocessing parameters are set by command line. We have prepared the following batch files that call the Preprocessor with certain predefined parameters as follows:

  • Preproc_rat.bat: Use this for rats (down sampling voxel size = 10 mm).
  • Preproc_rat(f).bat: Same as the previous one but it does not skip the data already preprocessed.
  • Preproc_mice.bat: Use this for mice (down sampling voxel size = 5 mm).
  • Preproc_mice(f).bat: Same as Preproc_rat(f), but for mice.

Example of a command line (Advanced):

> preprocessor.exe -f -2dvid -vsize 0.01 -camuse 0011
  • -f: no skip for the data preprocessed previously
  • -2dvid: generate 2D video files from RGBD data
  • -vsize: down sampling voxel size (in meters)
  • -camuse: selects cameras for the merging of the point clouds. 0=not use, 1=use.

In the example above, the merging points cloud is generated by using images from the camera #3 and #4.

Tracker usage

Starting the Tracker

Start the Tracker and select the xml file in the data folder that you want to analyze. Then, the following screen will appear.

_images/fig5-1.jpg

[Fig 5-1]

The windows on the screen are those you can launch by clicking the tabs in the menu Window. In addition, the skeletal models represented by connected spheres are shown in the Tracker screen. The 3D view can be controlled with the right button (R) by dragging the mouse as follows: R drag -> Rotate the view; R drag + Ctrl key -> Zoom in and out; R drag + Shift key -> translate the view.

The following “Player window” is used for searching a video frame and fitting execution.

_images/fig5-2.png

[Fig 5-2]

The numbers inside, and at the right side of the top slide bar, indicate the current # of the video frame and time, respectively. You can move the slide bar to search a specific video frame. By checking the box “Play” you can start the playback. The fitting of the skeleton models for pose estimation during playback is for the frames where it was not estimated. For the frames where it has already been estimated, the result of fitting is shown. The progress during the fitting process is displayed as “Tracking that has been completed…”. When starting the fitting process, you have to do it from the first frame and then continue with the next one. DO NOT skip frames during the fitting since it can result in an irreversible corrupted physic simulation.

Importing and setting parameters

Before starting the fitting, you need to set some parameters for the process. The default parameters (e.g., dimensions of a skeletal model) are optimized for rats. So, to analyze mice, import predefined parameters for mice (e.g., Track_params_mouse_v20171104.xml) from “File Menu > Import params”. The parameters can be displayed and modified with the “Model Editor window”, “Physics sim params window”, “ROI setting window”, and “Color filter window”.

_images/fig5-3.png

[Fig 5-3]

The “Model Editor window” is for the skeletal models. The slider on the top defines the number of animals (skeletal models). The scale of an animal can change depending on its age, sex, etc. So, you may need to change the scales using the other sliders in the red rectangle as shown in Fig 5-3. Detailed parameters outside the red rectangle (e.g., “Backbone Length”, “Radius for Physical Body”, etc.) don’t need to be changed frequently. After changing the parameters in the “Model Editor window”, click “Apply” button on the bottom, to apply the change.

The “Physics sim params window” is to parametrize the fitting algorithms, and usually it is necessary to change just some of them. The value of the “floor height” should be set to the height of the floor where the animal is standing in the global coordinate system. This parameter is important to constrain the movement of the model. The details of the parameters mentioned above are described in Chapter 9.

“ROI setting window” and “Color filter window” can be used to remove the points of non-animal objects. Removing such points will help fitting the skeleton models to the points cloud. In the color filtering, set the range of colors to include/exclude in the HSV color space (https://en.wikipedia.org/wiki/HSL_and_HSV). These parameters could be repeatedly used for the same experimental setup regarding rodent strain, age, apparatus, etc. You can re-use the parameters by exporting and then importing them from the “File menu > Export params” and “File menu > import params”, respectively.

Fitting

After the parameters have been set, start the fitting. Note that you cannot change the parameters during the fitting, otherwise the fitting will be re-initialized. In the first frame, manually set the initial position of the skeletal models. With shift key pressed, click two points from hip to head. You will see a line like the one in the figure below.

_images/fig5-4.png

[Fig 5-4]

If it is hard to know the direction of the subject in a 3D view, try using “2D video monitor window” as shown in Fig 5-1. Then, click the “Move” button in “Tracking tools window” to change the skeleton model to the correct place (Fig 5-5).

_images/fig5-5.png

[Fig 5-5]

The model is automatically fit from the initial position. If you find that the model was not fitted properly, manually adjust it by dragging body parts (sphere) of the model with the left mouse button. After placing the skeleton models, start the fitting process by enabling the “Play” check box in the “Player window”. The positions shown during the play are the result of the fitting process. As the fitting progresses, when you find an error in fitting, such as when a big part of a model is outside the 3D image, reversed direction, or swapping between animals (Fig 5-6), pause the fitting by disabling the “Play” check box.

_images/fig5-6.png

[Fig 5-6: an example of swapping error]

Then, by clicking “<<”, “<”, “>”, and “>>” buttons in the “Player window”, go back to the frame when the models start to get the error (Fig 5-7, middle). Then, manually adjust the models by dragging body parts (Fig 5-7, right) and re-start fitting from that frame by enabling “Play” check box.

_images/fig5-7.png

[Fig 5-7: correction of the swapping error shown in Fig. 5-6. Left: Actual mice positions, middle: before correction, right: after correction.]

If you find an error again, pause and fix it following the instructions described above. Repeat it and then progress with the fitting up to the last frames.

Hint

The Tracker has many functions to support this semi-automatic fitting process:

  • The speed of playback can be changed in “Option” in the “Player window”.
  • “Play” check box can be switched by pressing the space key.
  • By rotating the mouse wheel during the video pausing you can change the video frame.
  • By rotating the mouse wheel during playback it changes the playback speed.

Note

The fitting process must be done without skipping frames to get a successful result. Even if you don’t have an animal in the first frame, start from the first frame and not from the frame when the animal appears. For the fitting during the absence of animals, you can withdraw the model with “Withdraw” button in “Tracking tools window”. Note that the fitting process will be skipped if you move the frame using the slider on the top of the “Player window” or “<<”, “<”, “>”, or “>>” button below. When you restart the fitting after the manual correction, make sure that it restarts from the frame that you have corrected (check the progress of fitting in “Tracking has been completed…” display).

Saving and Exporting the fitting result

Once you have finished the fitting, you can re-check the result by playback or moving the slider in the “Player window”. The fitting result won’t be changed just by replaying the frames. You can save the fitting result from “File menu > Save result”. The result will be automatically loaded the next time you open the data. You can also reset the results to the one saved last time from “File menu > Load result”. This save/load function is also useful to interrupt the work. For the data analysis on the trajectories of body parts, you can export the csv format file (Fig 5-8) from “File menu > Export result”.

_images/fig5-8.png

[Fig 5-8]

Other software usage

Viewer

The software can be used as a viewer to check the recorded data but without the fitting functions. To open the data in Viewer, no pre-processing is necessary.

Data analysis scripts (Python / MATLAB)

These scripts are in preparation.

These scripts will be for analyzing the trajectory data (csv files exported from the Tracker) and the original point cloud data.

We encourage people to share their own scripts. Please let us know through the forum (https://groups.google.com/forum/#!forum/3dtracker) or the contact form (http://3dtracker.org/contact.html) if you have a useful code you would like to share. Also, you can find the old MATLAB scripts that were previously published with our paper, Matsumoto et al., 2013, (http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0078460#s6). Even though these old scripts cannot be directly used with the new data format, they would be a good starting point to write your own codes.

Trouble shooting

If you cannot find a specific topic associated to the 3DTracker or the solution for related some issue, ask your question in our forum (https://groups.google.com/forum/#!forum/3dtracker).

Recorder does not start (with R200 cameras)

The camera may not be recognized by the PC. Re-check the connection of the camera according to Chapter 2. If the problem persists, check whether the camera is listed in the “Device manager” (Fig 7-1).

_images/fig7-1.png

[Fig 7-1]

If the camera is not listed, make sure that “USB selective suspend” is disabled (https://www.techsupportalert.com/content/how-fix-annoying-windows-usb-problem.htm). If the camera is listed but the problem persists, uninstall the device of RGB camera and re-install it by clicking “scan for hardware changes” icon (Fig 7-2). It has been reported that a RGB camera driver error happens after “Windows 10 Creators Update”, and the steps above fix the problem. (https://github.com/IntelRealSense/librealsense/issues/481).

_images/fig7-2.png

[Fig 7-2]

Calibration does not work well

We suggest trying the following:

  • Change the infrared (IR) gain of the R200. If the camera is too close to the surface recorded or number of cameras is big, the IR reflection become too strong saturating the IR sensor. The data in the saturated part will be lost, interfering the pointer tracking during the calibration. Reduce “IR camera gain” in “Camera setting window” to avoid the saturation (Fig 7-3; see also Appendix for the camera mechanism).
_images/fig7-3.png

[Fig 7-3]

  • Check the area to scan with the pointer. Show “Camera monitor window” and make sure the pointer is always inside the view of all the cameras, and the distance between pointer and cameras is always more than 40 cm (R200 sensor can’t detect a depth < 40 cm). Also make sure the pointer light is not strongly reflected by the floor or other objects.
  • Change the room illumination and filters for the pointer detection. Adjusting the “Color filter”, “Outlier removal”, and room illumination will improve the quality of the pointer tracking.

Object or animals are not captured

Dropping frames during recording

Before you buy a faster PC, try the following:

Information for developers

Preparing the development environment

Use Visual Studio 2017 for the IDE.

First, download and install following libraries (You can download from our website at http://3dtracker.org/downloads.html):

  1. OpenCV 3.2.0 (library for image processing)
  2. PCL 1.8.0 (library for point cloud processing)
  3. Kinect for Windows SDK v1.8 (library for Kinect v1 camera and backward compatibility)
  4. 3DTracker Developer Tools (= librealsense, intel realsense camera library; Bullet physics, physics simulation library; freeglut, free opengl utility toolkit)
  5. (optional) NIDAQmx (National instrument device library; used for TTL signal)

Second, add the following folder to the environment variable “PATH”:

  • C:\Program Files\3DTrackerDevTools\bin
  • C:\opencv320\build\x64\vc14\bin
  • C:\Program Files\PCL 1.8.0\bin
  • C:\Program Files\OpenNI2\Redist

Programming Style

Basically follow https://google.github.io/styleguide/cppguide.html

More Information

Contact us through the forum (https://groups.google.com/forum/#!forum/3dtracker).

Appendix

R200 camera mechanism

ntel realsense R200 contains one RGB camera, two IR cameras, and one IR emitter. The IR emitter projects a random IR pattern to the objects (Fig 9-1, upper pictures). Distances from the camera to the objects’ surface are calculated using two IR camera images based on binocular disparity. As a result, a depth image (Fig 9-1, bottom picture), where each pixel in the image represents distance from the camera to the object surface, is generated.

_images/fig9-1.png

[Fig 9-1]

In 3DTracker software, a colored 3D point cloud is generated from the depth image and the RGB image captured by R200. Because R200 uses binocular disparity, the R200 cameras do not interfere each other. This is an excellent feature of R200 compared to the other 3D cameras (for example, ToF cameras, like Kinect v2) that severely interfere with each other. Although the R200 can work well in a multi-camera setup, if many of them are recording an object closely, the R200 would receive too much IR light causing some saturation (red circle in Fig 9-2).

_images/fig9-2.png

[Fig 9-2]

There is no binocular disparity in the saturated area, so the depth of the area cannot be calculated. By reducing the IR camera gain (in “Camera setting window” of the Recorder app), the saturation can be avoided. For the details of R200, see the Intel datasheet: https://www.intel.com/content/dam/support/us/en/documents/emerging-technologies/intel-realsense-technology/realsense-camera-r200-datasheet.pdf

Details of the parameters in the Tracker

Coming soon.

For reference see Matsumoto et al., 2013: http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0078460

Output files

The data of a recording session is saved as multiple files in the same folder. Each of the files contains data of different type and/or camera, as follows:

  • [Session_name].metadata.xml: metadata file, which stores recording configurations.
  • [Session_name].2dvideo.[N].avi: 2D RGB video of camera no. N.
  • [Session_name].camintrin.[N].bin: internal parameter of camera no. N.
  • [Session_name].rgbd.frame.[N].bin: RGBD data of camera no. N.
  • [Session_name].rgbd.ts.[N].txt: the timestamp of the RGBD data
  • [Session_name].pc.frame.[N].bin: Point cloud data of camera no. N.
  • [Session_name].pc.ts.[N].txt: the timestamp of the point cloud data
  • [Session_name].mrgpc.frame.bin: Merged point cloud data generated by Preprocessor
  • [Session_name].mrgpc.ts.txt: the timestamp of the merged point cloud data
  • [Session_name].trackparam.txt: Tracker parameters
  • [Session_name].trackresult.bin: Tracker result

Date last modified: 2018.01.22