|
Voodoo Camera Tracker: A tool for the integration of virtual and real scenesVersion 1.0.1 beta for Linux and WindowsCopyright (C) 2002-2009 Laboratorium für Informationstechnologie This non-commercial software tool is developed for research purpose at the Laboratorium für Informationstechnologie, University of Hannover. Permission is granted to any individual or institution to use, copy, and distribute this software, provided that this complete copyright and permission notice is maintained, intact, in all copies and supporting documentation. University of Hannover provides this software "as is" without express or implied warranty. Contents: [Overview] [Download] [Usage] [Troubleshooting] [Camera Model] [PNT Files] [Bug Report] OverviewThe Voodoo Camera Tracker estimates camera parameters and reconstructs a 3D scene from image sequences. The estimation algorithm offers a full automatic and robust solution to estimate camera parameters for video sequences. The results are useful for many applications like film production, 3D reconstruction, or video coding. The estimated parameters can be exported to the 3D animation packages: 3D Studio Max, Blender, Lightwave, Maya, and Softimage. The Voodoo Camera Tracker works very alike to commercial available camera trackers (also called match movers), e.g. 3D-Equalizer by Science-D-Visions, boujou by 2d3, Matchmover by RealViz , PFTrack by The Pixel Farm, SynthEyes by Andersson Technologies, or VooCAT by Scenespector Systems. Please consider buying a commercial product, if you need a camera tracker with professional support. The estimation method consist of five processing steps:
1) Automatic detection of feature pointsThe feature points are detected with sub-pixel accuracy using a corner detector. The detected feature points in the current image are marked with green crosses. Feature points that are detected as outliers during the processing are marked with red crosses.
2) Automatic correspondence analysisThe points can be matched from one image to the next by choosing matches which have the highest cross-correlation of image intensity for regions surrounding the points. The paths of the feature points are drawn in yellow here.
3) Outlier eliminationDue to misalignments or moving object in the scene some of the correspondences may be incorrect. To achieve a robust matching, a random sampling algorithm is employed to detect bad correspondences, called outliers. 4) Robust estimation of camera parametersThe camera parameters are estimated in an incremental fashion by using optimization technique applied to the good correspondences, called inlier. 5) Final refinement of the camera parametersFinally, a refinement step is applied to all camera parameters of the sequence. This step tries to distribute the estimation error evenly over the sequence. Afterwards, virtual objects can be integrated into the real image sequence.
DownloadThe Voodoo Camera Tracker is availabe for free download from http://www.digilab.uni-hannover.de as a binary file for Linux and Windows.UsageA typical camera tracking session includes the following steps
Main Tracking Window
After typing the command >voodoo the Main Tracking window is shown (Fig. 1).
Figure 1. Main Tracking Window The menu bar of this window has the following structure:
File → Load → SequenceBy choosing this item from the menu the Sequence Select window opens (Fig. 2). Figure 2. Sequence Select window You can load an image sequence that is stored in several image files. The Voodoo Camera Tracker is supporting Targa 2.0 (.tga), Portable Network Graphics (.png), and JPEG (*.jpg) image formats.
File → Load → Initial CameraBy choosing this item from the menu the Initial Camera Parameters window opens (Fig. 3). Figure 3. Initial Camera Parameter window You can set the initial camera parameters manually or load a camera file.
File → Load → CAHV CamerasFor each image of the sequence the Voodoo Camera Tracker holds one camera parameter set that is linked to this image. The parameter sets can be read from files which have the extension ".cam". The parameters are loaded for all the camera views in the image sequence by choosing this menu item. There must be one file for each image and the files must use the same numbering as the image sequence. File → Load → Feature PointsFor each image of the sequence the Voodoo Camera Tracker holds a set of feature points. The feature points can be read from files which have the extension ".pnt". The feature points are loaded for all images of the sequence by choosing this menu item. There must be one file for each image and the files must use the same numbering as the image sequence. File → Load → User PrimitivesThis menu item loads a user primitive in STL (stereolithography) format that can be placed into the scene by using the Modelling Tools. The Voodoo Camera Tracker can read STL files in binary or ASCII format. File → Save → 3D Studio Max (MAX Script)By choosing this item from the menu the Voodoo Camera Tracker exports the estimated camera parameters to the 3D animation package 3D Studio Max using a Max Script file. This file has the extension ".ms". To run the script go to [MAXScripts→Run Script] in 3D Studio Max. The image sequence can be displayed as an environment map in 3D Studio Max:
Figure 4. Export camera parameters to 3D Studio Max File → Save → Blender Python ScriptBy choosing this item from the menu the Voodoo Camera Tracker exports the estimated camera parameters to the 3D animation package Blender using a Python Script file. This file has the extension ".py". Load the python script into Blender's text editor and execute the script with ALT-P. Select the "voodoo_render_cam" camera and press Ctrl-Numpad 0 to activate it. To display the image sequence as a backbuffer in Blender, go to [Buttons Windows→Scene→Render Buttons→Output] and enter the file name of the image sequence in the "backbuf" field. A '#' at the end of the file name is replaced by the current (four-digit) frame number. We tested the import with Blender Version 2.42a. File → Save → LightWave 3DBy choosing this item from the menu the Voodoo Camera Tracker exports the estimated camera parameters to the 3D animation package Lightwave 3D using a LightWave Scene file. This file has the extension ".lws". To open the file, go to [File→Load→Load Scene] in Lightwave. The image sequence can be displayed as a background image in Lightwave:
File → Save → Maya ScriptBy choosing this item from the menu the Voodoo Camera Tracker exports the estimated camera parameters to the 3D animation package Maya using a Maya Script file. This file has the extension ".ma". To run the script go to [File→Open Scene] in Maya. The image sequence can be displayed as an environment map in Maya:
File → Save → Softimage 3.9 | XSI (3.0)By choosing this item from the menu the Voodoo Camera Tracker exports the estimated camera parameters to the 3D animation package Softimage 3D using XSI File Format V3.0. This file has the extension ".xsi" and can be imported to Softimage 3.9 or Softimage XSI. To import the file go to [Tools→Import→Objects→XSI Import] in Softimage 3.9 and in XSI go to [File→Import→Input .xsi].
Figure 5. Export camera parameters to Softimage 3.9 File → Save → TextfileThis menu item exports the estimated camera parameters and 3D feature points into a text file. We suggest you parse this file if you want to write your own import filter for an application that is currently not supported by the Voodoo Camera Tracker. File → Save → CAHV CamerasBy choosing this item from the menu the Saving Camera Files window opens (Fig. 6). Figure 6. Saving Camera Files window For each image of the sequence the Voodoo Camera Tracker holds one camera parameter set that is linked to this image. These parameters can be stored in a CAHV camera file (extention ".cam"). There is one file for each image.
File → Save → Feature PointsFor each image of the sequence the Voodoo Camera Tracker holds a set of 2D and 3D feature points. These feature points can be stored in ".pnt" files. There is one file for each image.
File → Save → User PrimitivesThis menu item saves user primitive that are placed into the scene with the Modelling Tools. File → ExitBy choosing this item from the menu the Voodoo Camera Tracker will be closed. All data, that is not saved, will be lost. View → ControlsBy choosing this item from the menu the Main Control Panel window opens (Fig. 7).
Figure 7. Main Control Panel window with selected Harris detector The camera parameter estimation (camera tracking) consists of five basic steps: Detection, Correspondence Analysis, Outlier Elimination, Estimation, and Final Estimation. For some steps the Voodoo Camera Tracker offers different algorithms. You are free to choose different combination of algorithms for every step. For example you can use the Harris , Foerstner, or Susan corner detector in the detection step. Some algorithms are in development state and therefore disabled in this beta version of the Voodoo Camera Tracker. If you are not familiar with the algorithms you can leave everything to the default configuration or play around with some parameters. In case of any trouble feel free to contact us for support. With the Main Control Panel you can choose the combination and parameter of the algorithms you want.
The default algorithm in the Correspondence Analysis step is the KLT Tracking method. It uses its own feature point detection algorithm and therefore you can not select an algorithm in the Detection step if the KLT Tracking method is enabled. The two most important parameters that sometimes need adjustment are modified with the "Max. Corners" and the "Win. Size for Local Max." text fields. The "Max. Corners" field limits the number of feature points per image. For sequences that are easy to track, you may want to reduce the number of feature points to speed up the computation. The "Win. Size for Local Max." can be used to evenly distribute the feature points in the image. A larger number increases the spread of the feature points. A nice distribution of feature points in the images is very important for good tracking results. Please also make sure that there are sufficient feature point tracks in the foreground and the background of the scene. You can also skip a step of the estimation process by clicking on the Enable/Skip field next to each step. The Voodoo Camera Tracker doesn't execute this step now, but uses the results already in memory for this step, if you press the Track button. This can be very useful, especially, if you edit, add, or remove feature points manually (see for Modelling Tools details). If you don't choose the Skip for the Detection and Correspondence Analysis step all your edited feature points will be overwritten. View → Draw OptionsBy choosing this item from the menu the Draw Options Deck window opens (Fig. 8).
Figure 8. Draw Options Deck window The settings in this window influence the 2D and 3D drawings which are overlayed in the actual image in the Main Tracking Window and in the 3D Scene Viewer. By clicking on the colored square, it is possible to select a color for each overlay. It is also possible to turn specific overlay elements on or off. In the following some important terms are defined
Fig. 9 shows an example of tracking results. Inlier are marked as green crosses and outlier as red crosses. The paths of feature points are drawn in yellow. One feature point is selected, marked as blue cross and the belonging search window is drawn in orange. Figure 9. Detected feature points and their paths View → Camera ParameterBy choosing this item from the menu the Camera Parameter window opens (Fig. 10.). Figure 10. Camera Parameter window This window has the same text fields as the Initial Camera window expect the Load button. For each image of the sequence the Voodoo Camera Tracker holds one camera parameter set that is linked to this image. By changing the displayed image in the Main Tracking Window the parameter shown in the Camera Parameter window change accordingly. Hence you have access to every parameter set of the sequence and can change the parameters according to your needs. Changes are applied to the stored camera parameter set by pressing the OK or Apply button. View → 3D Scene ViewerBy choosing this item from the menu the 3D Scene Viewer window opens (Fig. 11.). Figure 11. 3D Scene Viewer window In this viewer a 3D scene is displayed with a global 3D coordinate system. One unit of the coordinate system raster equals the focal length given by the actual camera parameter set. By pressing a key and then dragging the mouse while holding down a mouse button you can perform to following movements to explore the scene
The Options menu of the 3D Scene Viewer window contains a Draw Options window where you can choose whether a 3D object is drawn or not and what color is used for drawing. View → Modelling ToolsBy choosing this item from the menu the Modelling Tools toolbox is displayed (Fig. 12.). Figure 12. Modelling Tools toolbox With these modelling tools you can quickly place simple primitives into the scene and test your estimated camera parameters. First create a primitive by pressing a tool button in the Create toolbox. You can place a primitive easily by using the 2d anchor tool button. If the 2d anchor tool button is selected and you move the mouse over a vertex of the primitive, it will be highlighted (see Fig. 12). You can now move the vertex to the correct position in the image by clicking and dragging with the left mouse button. The translation, rotation and scale of the primitive is changed automatically as the vertex is moved. With the (T)-, (R)-, and (S)- buttons, which will appear when you select the 2d anchor tool, you can (de)activate the estimation of the primitive's (T)ranslation, (R)otation, and (S)cale parameters. To place the vertex more exactly you can use the zoom tool button. If you want to delete an object press the DELETE key. In Fig. 13 you can see the result of two fitted cubes. You can download this sequence from here, and try it yourself. Figure 13. Fitted Cubes With the selection tool button you can select the active primitive. A primitive can be selected by clicking on one of its vertices. The selection tool can also be used to select feature points. One feature point is selected by left click (see FPoint Editor) and multiple feature points are selected by right clicking and dragging the mouse over the feature points. To improve the estimation results it is often very useful to add manual feature point tracks. This can be done by using the create feature points button. Left click creates a new feature point at the position of the mouse pointer. The feature point is automatically selected. Now the user can go to the next consecutive image and add another feature point to the selected track by pressing the right mouse button. A correspondence between a selected feature point in the previous image and an existing feature point in the actual image can be generated by clicking on the existing feature point with the middle mouse button. Please note that the mouse wheel can be used to go quickly to the next image of the sequence. It is important to generate long tracks in the background as well as in the foreground and to distribute the feature points well in the images. Manually added feature points are marked with a golden rectangle. They will never be detected as outliers in the Outlier Elimination step. After you have added manual feature point tracks, you can try to estimate or re-estimate the camera parameters by pressing the Track button. Please make sure that you have selected Skip in the Detection and Correspondence Analysis step or all your edited feature points will be overwritten (see Main Control Panel for details). Sometimes it make also sense just to redo the Final Estimation step with the new manually added feature points. However, this should be only done, if your camera estimation result in memory is already very good and you added just a few manual points to make it perfect. If you want to execute the Final Estimation step on the camera estimation result in memory, Skip must be activated for all previous steps (see Main Control Panel for details). You should save the CAHV camera parameter and feature points before you try to add manual feature point tracks. If you have saved the CAHV camera parameters and feature points, you can always load them again and have another try if something went wrong. Another option to improve the estimation results is to remove wrong tracks that are not automatically detected as outliers. Use the selection tool to select a feature point and delete it using the FPoint Editor. If multiple feature points are selected by right click, you can delete them by pressing the DELETE key. After you have removed wrong feature point tracks you can try to improve the camera parameters by pressing the Track button and re-estimate the camera parameters (activate Skip in the Detection and Correspondence Analysis step). Fig. 14 shows a sequence with some manually added feature point tracks. The manually added feature points in the foreground are marked with golden rectangles around them. By manual adding and removing feature point tracks it is even possible to estimated camera parameters for footage with low image quality and many moving objects, like the one shown in Fig. 14 : "Trip Down Market Street Before the Fire (1905)", public domain footage supplied by archive.orgFigure 14. Manually added feature point tracks View → FPoint Track EditorDuring automatic tracking, feature point tracks often get lost because of perspective distortion of the tracked image patch, or because tracks in the background are cut by foreground objects, or because tracks get out of the camera's field of view and appear again later. The FPoint Track Editor allows to merge those disconnected tracks manually. This is usually done after you have estimated the camera parameters successfully and want to refine your solution. By choosing FPoint Track Editor from the menu the FPoint Track Editor window opens (Fig. 15). Figure 15. FPoint Track Editor window Select a feature point in the first part of the disconnected track using the selection tool button of the Modelling Tools. Then click the left From selection button in the FPoint Track Editor. The left two images in the FPoint Track Editor now display the images where the first part of the disconnected track starts and ends. Then move to another image and select a feature point in the second part of the disconnected track and click the right From selection button. Check carefully, if now in all four images the cross marks the same point in 3D and press the Merge button. After you have merged some of the broken tracks you can try to refine you estimation results by running a Final Estimation step (activate Skip in all estimation steps except the Final Estimation step and press the Track button). You should save the CAHV camera parameter and feature points before you try to refine the results manually. If you merge a wrong track this can have unpredictable results. If you have saved the CAHV camera parameter and feature points you can always load them again and have another try. If you press the Automatic Search button in the "Automatic Search for Merge Candidates" group, disconnected track candidates are searched automatically. Use the "Previous" and "Next" button to browse through the candidates and press the Merge button to accept the current candidate. Please note that it is usually not necessary to go through all the candidates. Just try to merge a few, e.g. 50, and check if this already sufficiently improves your results. The FPoint Track Editor can also be used to remove unwanted feature point tracks. Use the combo box in the "Remove FPoint Tracks" group to select what you want to remove. Then press the remove button. Help → HelpBy choosing this item from the menu this manual opens. Help → Notify LevelBy choosing this item from the menu the Notify Level window opens (Fig. 16). Figure 16. Notify Level window By choosing the notify level you can change the amount of information printed to the console, while the program is running. FPoint EditorFigure 17. FPoint Editor window The FPoint Editor window displays the parameters of a feature point. This window is displayed, if you select a feature point by using the "selection" tool button of the Modelling Tools. Troubleshooting
CAHV Camera ModelParameters of the CAHV camera model:(Cx, Cy, Cz) : CameraPosition [mm] (Ax, Ay, Az) : RotationAxis2 [unit vector] (Hx, Hy, Hz) : RotationAxis0 [pel] (including FocalLength, PixelSizeX, and Principal Point offset) (Vx, Vy, Vz) : RotationAxis1 [pel] (including FocalLength, PixelSizeY, and Principal Point offset) (K3, K5) : Radialdistortion K3 [1/(mm)^2] K5 [1/(mm)^4] (sx, sy) : PixelSize [mm/pel] (Width, Height) : ImageSize [pel]Internally used parameters: (ppx, ppy) : Principal Point offset [pel] f : Focal Length [mm] fov : Horizontal Field of View [degree] = (2*atan(0.5*Width*sx/f)*180/PI; (H0x, H0y, H0z) : RotationAxis0 [unit vector] (V0x, V0y, V0z) : RotationAxis1 [unit vector]Image coordinates and 3D coordinates: (x, y) : image coordinates [pel] (X, Y, Z) : 3D coordinates [mm]The projection of 3D coordinates in the camera image can be done by
[ x' ] = [ Hx Hy Hz ] [ 1 0 0 -Cx] [ X ]
[ y' ] = [ Vx Vy Vz ] [ 0 1 0 -Cy] [ Y ]
[ z' ] = [ Ax Ay Az ] [ 0 0 1 -Cz] [ Z ]
[ 1 ]
or
[ x' ] = [f/sx 0 ppx] [ H0x H0y H0z ] [ 1 0 0 -Cx] [ X ]
[ y' ] = [0 f/sy ppy] [ V0x V0y V0z ] [ 0 1 0 -Cy] [ Y ]
[ z' ] = [0 0 1 ] [ Ax Ay Az ] [ 0 0 1 -Cz] [ Z ]
[ 1 ]
then x = x'/z' and y = y'/z' , if the origin of the image coordinates is in the center of the image,or x = x'/z' + 0.5*(Width-1) and y = y'/z' + 0.5*(Height-1) , if the origin of the image coordinates is in the upper left corner.
PNT File FormatSaving the feature points with the Voodoo Camera Tracker generates one .pnt file for each input image.Each line in this text file represents a 2D feature point: x y manual type3d px py pz ident hasprev pcx pcy support x : x position of 2D feature point in that image y : y position of 2D feature point in that image manual : manual feature point (1 yes, 0 no) type3d : 0 for cartesian, 1 for spherical px : x position of 3D feature point py : y position of 3D feature point pz : z position of 3D feature point ident : unique identifier for a 3D feature point hasprev : has a previous feature point (1 yes, 0 no) pcx : x position of 2D feature point in the previous image pcy : y position of 2D feature point in the previous image support : inlier / outlier bitmaskIf the feature points have the same ident in different .pnt files they are on a common track. Bug ReportThe Voodoo Camera Tracker is developed for research purpose and has therefore less features than similar commercial products. Please consider buying a commercial product if you need a camera tracker with professional support. However, your experience report is important to us in order to improve the software according to your needs. Please send bugs and hints to Thorsten Thormählen (http://www.tnt.uni-hannover.de/~thormae) or Hellward Broszio (http://www.tnt.uni-hannover.de/~broszio). Do not hesitate to contact us in case of any trouble with this software.
|