Helga pipeline documentation¶
Welcome to the Helga pipeline documentation. Here you can find almost all answers to questions about how we do things within the Helga pipeline. It is the first place to look for answers if you have a questions. The second option is to ask me (Timm Wagener), you are very welcome to do that as often as possible, or write me an email.
Note
Please keep one thing in mind: A pipeline, or at least our pipeline, is not at first place a bunch of scripts that a TD hacks together that will automate this and that. A pipeline really is an agreement between all project workers, to meet certain standards that make life easier for everybody. All scripting and automatization comes after that and just builds on top of that!
Organization¶
Quick Guide¶
Note
Here is a quick cheat sheet of how we be rollin. In 95% of all cases you do the right thing when you follow these simple rules.
- Communicate!
If you are unsure, dont do it. Talk to the person that knows!!!
If you dont know, and nobody is there to tell you: Use your rnd folder.
- Use the naming convention
To let people will know who is responsible, what is the latest version etc. Read about it here
- Your work is not done when your work is done!
Your work is done when you correctly gave it to the coworker who uses it next (Publishing/Copying/Overwriting).
- Workspace and Publishing
This is the system that builds upon the former rule. Work in your workspace and give it to your coworkers when its ready (publish). Read more about it here. Much of the work your coworkers do depends on you being clean and organized (rig references for animators, texture references for shading artists, matte painting references for comp artists).
Work clean and dont rush! If you dont work clean, you cause somebody else more work.
Never reference/use something from a work directory
Naming Convention¶
Here you can find a quick reference for the naming convention we use.
Note
This naming convention holds true for most of our pipeline. Please try to match it as close as possible. Small errors count! Think of it as right not if a human can understand your versioning but if an algorythm could successfully parse your files and do something meaningfull without errors.
For work files it usually goes like :
ulfbert_model_a_0001_mh.mb
ulfbert_sculpt_d_0012_mh.ztl
helga_rig_base_c_0023_tw.mb
snorri_concept_f_0012_ms.psd
Quick explanation¶
- ulfbert_model:
Name that explains what file we are dealing with. Could also be snorri_base_model or king_larry_the_third_model_from_hell, just make sure people know the item that is versioned here.
- a:
Wide iteration letter that is for large changes and is a landmark for you. You usually know (at least when you open a file) why you upversioned the letter
- 0001:
Version number. Always has four digits.
- mh:
Modifier initials of the person that saved this file.
Chain them all together and you have a beautifully explanatory naming convention that helps everybody understand what he is dealing with.
Published files¶
Published files are a little bit different in that they just omit the versioning suffixes. This makes sense when you think about them as the current state of your work you made available to everybody.
Here is an example for the ulfbert rig :
Work folder¶
Here is the file how the rigger names and versiones it up
ulfbert_rig_body_c_0023_ah.mb
ulfbert_rig_body_c_0024_ah.mb
ulfbert_rig_body_d_0025_ah.mb
ulfbert_rig_body_e_0026_ah.mb
Work and Publish¶
Our system of work and publish explained. It is really as simple as copying and overwriting your work at the right locations, so dont be afraid ;)
Separate work and publish locations¶
This is just the simple principle that you manage what you give to production. Fellow coworkers never see your messy workspaces and only grab the content that you have published for them to use.
Important
If the object you want to use is not published, then dont just use a version from the responsible persons workspace. Instead ask the person for the content.
In many locations you will find a work folder which might contain the same folderstructure where itself is in for example :
3d/maya/scenes/assets/ <-- This is where people USE your work
chars
props
cameras
...
**work**
3d/maya/scenes/assets/work <-- This is where people work
chars
props
cameras
...
Here is an example for concept artists, to show that this principle is not limited to 3d :
2d/concept_art <-- This is where people watch work
chars/
ulfbert/
ulfbert_concept_hair <-- People go here to WATCH the latest ulfbert hair concept
props
...
**work**
2d/concept_art/work <-- This is where people work
chars
ulfbert/
ulfbert_concept_hair_a_0001_mh.psd <-- People go here to CREATE the latest ulfbert hair concept
ulfbert_concept_hair_b_0002_mh.psd
...
props
...
Pipeline¶
Here you can find some infographics to explain our pipeline visually.
Assets and shots pipeline¶
Photoscan pipeline¶
Fur workflow¶
Caching¶
- Load Alembic from animation or simulation (both are identical) and use the character_fur HDA to create, simulate and tweak the fur.
- Cache the complete fur to a seperate file. This is the distilled and only output for fur. (All of the creative part of the fur process happens in a separate fur hip file, analog to the cloth or other effects files).
Rendering¶
- Load the character_fur_render HDA and point it to the cache. (Or to the cache directory if several caches are needed for several hair systems). If not all needed caches exist, the fur rendering will fail. In render files will be NO tweaking on the fur or artistry on simulation/animation, only shaders will be tweakied there.
- Load character_fur_shading HDA and plug it in material slot of character_fur_render HDA.
Required Digital Assets¶
- character_fur HDA
Provides all the functionality to create/simulate/tweak and cache the fur. This is the motor.
Note
Needed functionality:
- Read standardized Alembic from Animation/Simulation and create fur on it.
- Ability to art direct the fur based on directors oppinion
- Simulate fur in order to account for intersection with cloth
- Cache the fur.
All of this happens in a seperate fur file for each shot done by the fur artist.
- character_fur_shading HDA
Analog to usual shading assets. An HDA that just contains one or more materials to be plugged in the material slot(s) of the character_fur_render HDA.
- character_fur_render HDA
HDA that wraps the character_fur HDA, hides all the complexity of the fur itself and exposes all the attributes needed for rendering.
Note
Needed functionality:
- Material slots which allow to pick the materials from the character_fur_shading HDA or any other hair materials
- Pick the cache done by the fur artist
- Enable visibility for rendering and viewport
The character_fur_render HDA wraps the character_fur HDA for DRY reasons and to provide a stripped interface that hides the complexity for shading artists. If there is no cache, the fur cannot be rendered.
Standards¶
Formats and Standards¶
Note
Here you can find a list of formats and standards that we use in our production.
Textures¶
Painted textures (color, specular, reflection, fur maps etc..... almost everything) | .tga |
Photoscan textures (easier because .jpg doesnt have an alpha channel) | .jpg |
Data maps, 32bit textures (displacement, etc.) | .exr |
Textures for rendering (32bit and 8bit.....basically all) | .rat |
Rendering¶
.exr | Half-float (16bit), zip-scanline, multichannel, Auto-data window, linear (Gamma 1.0) |
3d¶
Maya | .mb |
Cache¶
Geometry (Rigs, props etc.) | .abc |
Cameras (Shotcam....) | .abc |
Locator | .abc |
Fur (Guide curves are always cached before rendering). | .bgeo |
Houdini Variables¶
Note
Here is a quick overview of all the specific variables that can be used in Houdini. They should always be used instead of a hard wired path. They will expand before an .ifd is generated so they should be working with the farm properly.
All helga variables are prefixed with HELGA, if they arent, they are not specific to the helga project.
Note
This list is subject to constant updates.
Shots¶
$HELGA_SHOT_NAME | Name of the current shot | 030_establishertavern, 240_kick, etc. |
This variable is added by the pipeline on startup if it doesnt exist. However it is NOT given a value. The correct value for the current shot has to be set by the user. Please see below for examples.
Warning
Applications¶
$HELGA_HOUDINI_EXE | Path to pipeline Houdini | C:/symlinks/houdini/13.0.260/hfs/bin/houdinifx.exe |
$HELGA_MAYA_EXE | Path to pipeline Maya | C:/symlinks/maya/maya2014x64/bin/maya.exe |
$HELGA_NUKE_EXE | Path to pipeline Nuke | D:/tools/Nuke8.0v1/Nuke8.0.exe |
Pipeline¶
$HELGA_PIPELINE_BASE_PATH | Root path of server | //bigfoot/grimmhelga |
$HELGA_PIPELINE_FLAVOUR | Currently active pipeline flavour. Could be deploy or sandbox etc. | deploy, sandbox etc. |
Houdini¶
$HELGA_SCRIPTS_BASE_PATH | Python package base path. From here you can import helga. | //bigfoot/grimmhelga/Production/scripts/deploy/helga |
$HELGA_HDRI_PATH | Path for all HDRIs | $HELGA_PIPELINE_BASE_PATH + /Production/2d/hdri |
$HELGA_OTL_PATH | Default path for helga otls | $HELGA_PIPELINE_BASE_PATH + /Production/3d/maya/scenes/assets/otls |
$HELGA_FUR_PATH | Path for fur caches | $HELGA_PIPELINE_BASE_PATH + /Production/3d/maya/cache/fur |
$HELGA_ALEMBIC_PATH | Path for all abc caches used in lighting, fur and sim. | $HELGA_PIPELINE_BASE_PATH + /Production/3d/maya/cache/alembic |
$HELGA_TEXTURES_PATH | Path for all textures used in the helga project. | $HELGA_PIPELINE_BASE_PATH + /Production/3d/maya/scenes/assets/textures |
$HELGA_RENDER_PATH | Root Path where all images are rendered to from 3d. | $HELGA_PIPELINE_BASE_PATH + /Production/3d/maya/images |
Checklists¶
Mesh checklist¶
Note
Don’t let Marco H. trick you.... here’s what to look out for ;)
- Normals facing correct ?
Probably by default do Normals>Conform and Normals>Set to Face on all geo in Maya.
- Is the geo frozen (Translate, Rotate, Scale) and the pivots of all objects are at (0,0,0) in worldspace?
This ensures all Alembics will be at the right position under the world later on and its also cleaner when rigging. In Maya you can run this:
import pymel.core as pm selected_nodes = pm.ls(sl = True, fl = True) for selected_node in selected_nodes: #freeze pm.makeIdentity(selected_node, a = True) #reset pivot pm.xform(selected_node, piv = (0,0,0))
- Doublesided and Opposite turned off?
This ensures that all faces point to the desired direction for all renderers without custom correction. In Maya you can do this in the Attribute Spreadsheet under the Render tab. (The first two tabs)
- No empty UV Sets floating around and all primary UV Sets are enabled?
In case there are empty UV sets it makes sense to delete them. Also for some reason Maya likes to have its primary UV set named map1. While this is not always necessary, its good to follow this convention if possible.
- No overlapping UVs on all the primary UV sets?
Maybe sometimes there might be reasons to do otherwise but in general its a good idea to have UV sets not overlapping, so that follicles or other rigging nodes that are placed on the surface based on UVs work correct.
- UVs have the correct winding order? (Are the UV shells blue instead of red?)
This happens through flipping and freezing of geometry for example. Can be fixed with Edit UVs>Flip
- Do objects have the right scale?
Default units in Maya are cm.
- Only geo nodes in scene?
No history, random materials (for example from .obj import) or other nodes should pollute the scene. All geo should have the default Lambert material assigned.
- Geometry usefully grouped?
- grp_geo
- grp_body
grp_left_leg...
- All geometry that is actively simulated (cloth etc.) needs to be all quads and tris!
Collision geometry is fine with NGons. When in doubt ask Johannes.
Animation checklist¶
Note
Here are some important things for the animators amongst us. Since your output is used in simulation, lighting, shading etc. please pay attention to these points, to ensure everything is running smoothly through the pipe.
Simulation Preroll¶
In order for the extensive cloth sim, that we want to put on top of the helga characters, there is some pre-roll time necessary. Trust me, it’s not rocket science, here’s how we wanna do it.
There are 2 steps that come before the character animation that are needed for the sim dudes.
- Frame: 901 - 951
Character is in T-Pose. The cloth settles on the T-Pose character.
- Frame: 951 - 1001
On frame 951 there is a keyframe of the T-Pose. From here the character morphs linearly into the start pose. Here the cloth gets ready for the real action.
- Frame: 1001
This is the first keyframe of the character animation.
We choose frame 1001 as the first keyframe of animation because it’s a round 1000 frames offset from the usual frame 1.
Photogrammetry checklist¶
Note
Here i note the settings which seem to provide a good quality/time ratio. (For final solves). They are based on the first days of testing and finding out what works/doesnt work.
Solve settings (final)¶
Set the settings in the batch process dialog to the following. Before the aligning of chunks (second step) it makes sense to check the solved chunks and disable those that dont work before aligning and merging chunks.
Note
Import as multiple chunks
Align Photos: High
- Align Chunks:
- chunks count < 10: High
- chunks count > 10: Medium
- Build dense cloud:
- chunks count < 10: High
- chunks count > 10: Medium
- Mesh:
- Face count: Custom (2.000.000 polies).
- Texture:
- Size: 8192²
- Format: .jpg
Machine list¶
The following machines have a Photoscan license:
- Pegasus (Manuel)
- Gnocchi (Timm)
- Maccheroni (Timm)
- Adonis (Johannes)
- Fusilli (Silke)
- Gemelli (Nicole)
Asset checklist¶
Note
In order for the smooth, customized export of assets from animation to lighting/shading, sim and everybody else, the assets need a little treatment before going in production. Here’s a little list of things to take care of. It is advisable (although not needed) to use the AssetManager to automate a lot of these steps in Maya.
- Asset has blank prop or char metadata node ?
This tells the asset manager which categories you want to export. The node options are left blank, because they need to be filled in the scene as a local reference edit.
(Shot metadata nodes are embedded in the shot files, and never belong into an asset).
- Export Attributes attached (proxy, rendergeo, locator...)?
This tells the asset manager what the content of the categories is. Do all the export objects have the correct attributes assigned. The following attributes are typical:
- helga_proxy
- helga_rendergeo
- helga_locator
- helga_highpoly_rendergeo <– Points to the highpoly render version of the asset.
These attributes are always assigned on the transform nodes of the objects in Maya. No attributes are put on shape nodes by default. (Although its perfectly fine by artists to do so).
The asset manager tool provides an interface to add or remove those attributes on selected nodes:
Only objects with export attributes will be considered for the export
- helga_highpoly_rendergeo attribute on locator set?
The helga_highpoly_rendergeo attr. is filled out on the asset before referencing. It points to the highpoly render replacement geometry, usually an alembic file located in cache/alembic/highpoly_rendergeo.
- Does the asset have a material?
Make sure the textures are lightweight and published in the textures/props/prop_xy directory.
Does the asset have a rig, and are the joints and locator hidden?
- Did you check the asset with the AssetManager after publishing?
Always be the first to reference in your newly published asset and check if it performs with the AssetManager.
I am a¶
I am a ...¶
Here you can quickly find the important ins & outs sorted for your specific task.
I am a texture painter¶
Note
Where to put your Mari, Mudbox projects or PS files?¶
Please work on your textures in this folderY:/Production/3d/maya/scenes/assets/work/chars/CHARACTERNAME/texturesY:/Production/3d/maya/scenes/assets/work/props/PROPNAME/texturesPlease version up like this:ulfbert_textures_a_0001_mh.mraulfbert_textures_a_0002_mh.mraulfbert_textures_b_0003_mh.mra...chimney_textures_e_0005_mh.mudchimney_textures_f_0006_mh.mud
Where to put the textures for the shading artist to use?¶
I am a rigger¶
Rigging artist information
I am a modeler¶
Modeler information
I am a concept artist¶
Concept Artist information
I am a animator¶
Hey animation crew, here is a quick overview of the what, where and how within the *helga* animation pipeline. Dont be scared, we try to make life for you as easy as possible! Please risk a quick look at these points, and in a spin you should be ready to animate :D
Who can help if there are problems?¶
Dont be afraid to ask as often as possible! Remember: Some TDs look more grim than they really are ;D
Where do i get those rigs?¶
You can find them here:Y:/Production/3d/maya/scenes/assets/charsImportant: If the rigs are not there or do not behave as expected pleasedont fix this yourself or grab them from another location.Instead please ask our riggers Hanna andArash, since its their responsibilityto supply you with correct working material (and also they are sick experts in the biz of rigging...much more than u and me)
Where are the shots?¶
Y:/Production/3d/maya/scenes/shots
Why is everything an asset and not just a model or rig?¶
Assets are nothing special, they are just models and rigs that meet specialcriteria. An asset has metadata for example to guarantee that it willexport correctly for the lighting dudes.The general idea is, that everything you animate (Set, Characters and Props)is an asset so a smooth functioning is guaranteed.
I want an additional asset in my shot. How?¶
You need another beer bottle or an empty glass to get the perfect framing?Its totally fine to reference another asset in your shot, as long as yougrab them from here:Y:/Production/3d/maya/scenes/assets/propsThose assets have been messed with by a TD and are guaranteed to exportcorrectly to lighting, simulation etc.If the asset you need isnt there, just ask me and i will try to add it.
Preroll?¶
After animation is approved, the simulation department (Johannes)will do its thing.In order for them to do a good job, you need to supply them with a so calledResting Pose. The resting pose basically is a T-Pose that blends linear intoyour first frame of animation. You can read more about it here.
Pipeline¶
API Overview¶
Foreword¶
First let me say that API is probably a little misleading since the helga package is mostly a collection of custom tools written in Python, which are independent of each other (like helga.nuke.reconstruction.renderReconstructVRay). But there might be modules in the future that are ment to be exclusively building blocks for programmers.
Organization¶
Our pipeline is split up into several kinds of packages that import from a logical hierarchy. We can add packages at any depth in the hierarchy at any time.
DCC packages¶
Category packages¶
Please note the packages named after contributors. Packages without a name are assumed to be completely written by me (Timm Wagener). If you want to contribute just ask and you will get your own category package.
Warning
Please never contribute outside of your own category package!
Work and Deploy¶
There are two Helga packages located under:
production/scripts/work:
Here is where we work and edit our scripts. You are expected to always work here and copy your stuff over to deploy when ready. If you work and edit in deploy to save the copy step, you will sooner or later loose work, since i will often delete and recopy the whole helga package.
production/scripts/deploy:
Here is where the DCCs use our scripts and their compilers do the byte compilation (create .pyc files)
This is basically source control for the poor man, but since the helga package was initially ment to be exclusively my own sandbox we have to deal with that inconvenience.
Coding Convention¶
If you want to contribute to our pipeline, please let me know. There is a place for these efforts and i will try to support you with that. In return you are expected to follow our coding standards in order for your code to integrate consistently with the codebase and make it easier for everybody.
Our coding convention in one sentence:
Warning
The current codebase contains large portions of legacy code. While still functional it was created before the coding conventions where active. This will be refactored over time.
Important PEPs¶
There are a lot of PEPs. Some are more important than others. You are expected to read and use them. Often times in this document, i will just refer to the appropiate place in some PEP.
Variables, Classes, Modules, Packages etc.¶
For naming of objects in Python we use the Google naming convention. (PEP 8 does not recommend any naming).
module_name, package_name, ClassName, method_name, ExceptionName, function_name,
GLOBAL_CONSTANT_NAME, global_var_name, instance_var_name, function_parameter_name,
local_var_name.
You can find the full documentation here.
Running on a different OS¶
Note
The helga pipeline is developed/maintained foremost on the windows platform. However there might be the need sometimes to switch to a different operating system like Linux. For these cases you can find a quick guide here how to start the DCCs in the correct environment, but keep in mind, that support for operating systems other than windows at the moment is very experimental.
Warning
This guide and method are considered temporary and might become deprecated once there is time for a refactoring of the HelgaLauncher tool that is used under windows.
Tested on¶
- CentOS7
- Houdini: Yes
- Nuke: No
- Maya: Untested
Step by step guide¶
Please note that this guide is geared towards TDs or people that are a little familiar with git or Python, since there is. a little effort involved in making it run outside of how it is ment to be run. 6 little steps to take on your own risk :D
- Temporary clone the helga git repo
Clone the helga repo to a temporary location. Just navigate to some temp folder and execute this snippet in your git enabled shell:
git clone https://github.com/timmwagener/helga.git helga
This should create a folder named helga at your current path and clone the helga repository in it.
- Helga folder structure
Now in your temporary helga repo navigate to helga/general/setup/direct_access/supplementary_material/folder_structure. There you will find a zip file called helga.zip, that contains the helga folder structure. Extract this zip to where you want the helga pipeline and project to be located. PS: The helga folder structure is based on the Maya project structure which might be unfamiliar to non Maya users.
- The real clone - work repository
Now its time to do the real clone of the helga repo and fit it into the new folder structure. Navigate to your_root/Production/scripts/work and execute
git clone https://github.com/timmwagener/helga.git helga
there again. This is the repo you will keep up-to-date and maintain and always copy into sandbox or deploy when you want to publish changes to production.
- Start procedure for Houdini, Maya & Nuke
Now its time to adjust the startup module for your dccs (Supported are Maya 2014, Nuke 8.x and Houdini 13.0.xxx). Please go to Production/scripts/sandbox/helga/helga/general/setup/direct_access/yaml where you will find a file called pipeline_base_data.yaml. Open it with a text editor, for example sublime text or notepad and adjust the following values:
- PIPELINE_BASE_PATH: “path/to/where/Production/and/Organization/folders/are/under”
- PIPELINE_FLAVOUR: “deploy” <– Pick either sandbox or deploy. This is the folder name where the script environment you want to use is under.
- MAYA_EXE: “path/to/maya.exe”
- NUKE_EXE: “path/to/Nuke8.0.exe”
- HOUDINI_EXE: “path/to/houdinifx.exe” (or houdini.exe)
Save and you are good.
- Pipeline repo production copies - sandbox and deploy
Those two fellas could also be named experimental and stable, beta or gold......you get the idea. Just go and copy the whole helga folder, in which you just did modifications, and __init__.py file from work into sandbox and deploy.
- Finally
Test if the application runs correctly. Navigate to deploy|sandbox/helga/helga/general/setup/direct_access and type the following in a python enabled shell:
#Houdini: python -c 'import direct_access;reload(direct_access);direct_access.run("houdini")' #Maya: python -c 'import direct_access;reload(direct_access);direct_access.run("maya")' #Nuke: python -c 'import direct_access;reload(direct_access);direct_access.run("nuke")'
If everything went well, your DCC should start within the helga pipeline environment (either sandbox or deploy)
Hopefully that was not too confusing ;) Please remember that you now have the pathes and setup in place, the structure if you will, but you still need a lot of content like .otls for example. In your own production you would of course populate the pipeline with your own scripts, plugins, digital assets, gizmos etc.
Good luck :D
API Documentation¶
helga.houdini | helga.houdini package |
helga.maya | helga.maya package |
helga.nuke | helga.nuke package |
helga.general | helga.general package |