alternate text

Helga pipeline documentation

Welcome to the Helga pipeline documentation. Here you can find almost all answers to questions about how we do things within the Helga pipeline. It is the first place to look for answers if you have a questions. The second option is to ask me (Timm Wagener), you are very welcome to do that as often as possible, or write me an email.

Note

Please keep one thing in mind: A pipeline, or at least our pipeline, is not at first place a bunch of scripts that a TD hacks together that will automate this and that. A pipeline really is an agreement between all project workers, to meet certain standards that make life easier for everybody. All scripting and automatization comes after that and just builds on top of that!

Organization

Quick Guide

Note

Here is a quick cheat sheet of how we be rollin. In 95% of all cases you do the right thing when you follow these simple rules.


  1. Communicate!

    If you are unsure, dont do it. Talk to the person that knows!!!

  2. If you dont know, and nobody is there to tell you: Use your rnd folder.

  3. Use the naming convention

    To let people will know who is responsible, what is the latest version etc. Read about it here

  4. Your work is not done when your work is done!

    Your work is done when you correctly gave it to the coworker who uses it next (Publishing/Copying/Overwriting).

  5. Workspace and Publishing

    This is the system that builds upon the former rule. Work in your workspace and give it to your coworkers when its ready (publish). Read more about it here. Much of the work your coworkers do depends on you being clean and organized (rig references for animators, texture references for shading artists, matte painting references for comp artists).

  6. Work clean and dont rush! If you dont work clean, you cause somebody else more work.

  7. Never reference/use something from a work directory

Naming Convention

Here you can find a quick reference for the naming convention we use.

Note

This naming convention holds true for most of our pipeline. Please try to match it as close as possible. Small errors count! Think of it as right not if a human can understand your versioning but if an algorythm could successfully parse your files and do something meaningfull without errors.


For work files it usually goes like :

ulfbert_model_a_0001_mh.mb
ulfbert_sculpt_d_0012_mh.ztl
helga_rig_base_c_0023_tw.mb
snorri_concept_f_0012_ms.psd

Quick explanation

  • ulfbert_model:

    Name that explains what file we are dealing with. Could also be snorri_base_model or king_larry_the_third_model_from_hell, just make sure people know the item that is versioned here.

  • a:

    Wide iteration letter that is for large changes and is a landmark for you. You usually know (at least when you open a file) why you upversioned the letter

  • 0001:

    Version number. Always has four digits.

  • mh:

    Modifier initials of the person that saved this file.

Chain them all together and you have a beautifully explanatory naming convention that helps everybody understand what he is dealing with.


Published files

Published files are a little bit different in that they just omit the versioning suffixes. This makes sense when you think about them as the current state of your work you made available to everybody.

Here is an example for the ulfbert rig :

Work folder

Here is the file how the rigger names and versiones it up

ulfbert_rig_body_c_0023_ah.mb
ulfbert_rig_body_c_0024_ah.mb
ulfbert_rig_body_d_0025_ah.mb
ulfbert_rig_body_e_0026_ah.mb
Publish folder

Here is the published file. This file is used by all animators as a reference to do the animation so by overriding it, the rigger automatically updates the whole production.

ulfbert.mb

Read more about our work and publish system here

Work and Publish

Our system of work and publish explained. It is really as simple as copying and overwriting your work at the right locations, so dont be afraid ;)


Separate work and publish locations

This is just the simple principle that you manage what you give to production. Fellow coworkers never see your messy workspaces and only grab the content that you have published for them to use.

Important

If the object you want to use is not published, then dont just use a version from the responsible persons workspace. Instead ask the person for the content.

In many locations you will find a work folder which might contain the same folderstructure where itself is in for example :

3d/maya/scenes/assets/ <-- This is where people USE your work
        chars
        props
        cameras
        ...
        **work**

3d/maya/scenes/assets/work <-- This is where people work
        chars
        props
        cameras
        ...

Here is an example for concept artists, to show that this principle is not limited to 3d :

2d/concept_art <-- This is where people watch work
        chars/
                ulfbert/
                        ulfbert_concept_hair <-- People go here to WATCH the latest ulfbert hair concept
        props
        ...
        **work**

2d/concept_art/work <-- This is where people work
        chars
                ulfbert/
                        ulfbert_concept_hair_a_0001_mh.psd <-- People go here to CREATE the latest ulfbert hair concept
                        ulfbert_concept_hair_b_0002_mh.psd
                        ...
        props
        ...

Pipeline

Here you can find some infographics to explain our pipeline visually.


Assets and shots pipeline

Assets and shots pipeline chart

Photoscan pipeline

Photoscan pipeline chart

Fur workflow

Caching

  1. Load Alembic from animation or simulation (both are identical) and use the character_fur HDA to create, simulate and tweak the fur.
  2. Cache the complete fur to a seperate file. This is the distilled and only output for fur. (All of the creative part of the fur process happens in a separate fur hip file, analog to the cloth or other effects files).

Rendering

  1. Load the character_fur_render HDA and point it to the cache. (Or to the cache directory if several caches are needed for several hair systems). If not all needed caches exist, the fur rendering will fail. In render files will be NO tweaking on the fur or artistry on simulation/animation, only shaders will be tweakied there.
  2. Load character_fur_shading HDA and plug it in material slot of character_fur_render HDA.

Required Digital Assets

  1. character_fur HDA

Provides all the functionality to create/simulate/tweak and cache the fur. This is the motor.

Note

Needed functionality:

  1. Read standardized Alembic from Animation/Simulation and create fur on it.
  2. Ability to art direct the fur based on directors oppinion
  3. Simulate fur in order to account for intersection with cloth
  4. Cache the fur.

All of this happens in a seperate fur file for each shot done by the fur artist.

  1. character_fur_shading HDA

Analog to usual shading assets. An HDA that just contains one or more materials to be plugged in the material slot(s) of the character_fur_render HDA.

  1. character_fur_render HDA

HDA that wraps the character_fur HDA, hides all the complexity of the fur itself and exposes all the attributes needed for rendering.

Note

Needed functionality:

  1. Material slots which allow to pick the materials from the character_fur_shading HDA or any other hair materials
  2. Pick the cache done by the fur artist
  3. Enable visibility for rendering and viewport

The character_fur_render HDA wraps the character_fur HDA for DRY reasons and to provide a stripped interface that hides the complexity for shading artists. If there is no cache, the fur cannot be rendered.

Standards

Formats and Standards

Note

Here you can find a list of formats and standards that we use in our production.


Textures

Painted textures (color, specular, reflection, fur maps etc..... almost everything) .tga
Photoscan textures (easier because .jpg doesnt have an alpha channel) .jpg
Data maps, 32bit textures (displacement, etc.) .exr
Textures for rendering (32bit and 8bit.....basically all) .rat

Rendering

.exr Half-float (16bit), zip-scanline, multichannel, Auto-data window, linear (Gamma 1.0)

3d

Maya .mb

Cache

Geometry (Rigs, props etc.) .abc
Cameras (Shotcam....) .abc
Locator .abc
Fur (Guide curves are always cached before rendering). .bgeo

Houdini Variables

Note

Here is a quick overview of all the specific variables that can be used in Houdini. They should always be used instead of a hard wired path. They will expand before an .ifd is generated so they should be working with the farm properly.

All helga variables are prefixed with HELGA, if they arent, they are not specific to the helga project.

Note

This list is subject to constant updates.


How To?

Here’s how its used in Houdini. Its simple and comfortable.

Houdini env. Variables expand example.

Shots

$HELGA_SHOT_NAME Name of the current shot 030_establishertavern, 240_kick, etc.

This variable is added by the pipeline on startup if it doesnt exist. However it is NOT given a value. The correct value for the current shot has to be set by the user. Please see below for examples.

Warning

When importing Alembics (characters, shots, cameras) please always have your path starting with:
$HELGA_ALEMBIC_PATH/$HELGA_SHOT_NAME/
For example:
$HELGA_ALEMBIC_PATH/$HELGA_SHOT_NAME/props/
$HELGA_ALEMBIC_PATH/$HELGA_SHOT_NAME/cameras/shot_cam.abc
$HELGA_ALEMBIC_PATH/$HELGA_SHOT_NAME/chars/ulfbert_rendergeo.abc

Applications

$HELGA_HOUDINI_EXE Path to pipeline Houdini C:/symlinks/houdini/13.0.260/hfs/bin/houdinifx.exe
$HELGA_MAYA_EXE Path to pipeline Maya C:/symlinks/maya/maya2014x64/bin/maya.exe
$HELGA_NUKE_EXE Path to pipeline Nuke D:/tools/Nuke8.0v1/Nuke8.0.exe

Pipeline

$HELGA_PIPELINE_BASE_PATH Root path of server //bigfoot/grimmhelga
$HELGA_PIPELINE_FLAVOUR Currently active pipeline flavour. Could be deploy or sandbox etc. deploy, sandbox etc.

Houdini

$HELGA_SCRIPTS_BASE_PATH Python package base path. From here you can import helga. //bigfoot/grimmhelga/Production/scripts/deploy/helga
$HELGA_HDRI_PATH Path for all HDRIs $HELGA_PIPELINE_BASE_PATH + /Production/2d/hdri
$HELGA_OTL_PATH Default path for helga otls $HELGA_PIPELINE_BASE_PATH + /Production/3d/maya/scenes/assets/otls
$HELGA_FUR_PATH Path for fur caches $HELGA_PIPELINE_BASE_PATH + /Production/3d/maya/cache/fur
$HELGA_ALEMBIC_PATH Path for all abc caches used in lighting, fur and sim. $HELGA_PIPELINE_BASE_PATH + /Production/3d/maya/cache/alembic
$HELGA_TEXTURES_PATH Path for all textures used in the helga project. $HELGA_PIPELINE_BASE_PATH + /Production/3d/maya/scenes/assets/textures
$HELGA_RENDER_PATH Root Path where all images are rendered to from 3d. $HELGA_PIPELINE_BASE_PATH + /Production/3d/maya/images

Checklists

Mesh checklist

Note

Don’t let Marco H. trick you.... here’s what to look out for ;)


  1. Normals facing correct ?

    Probably by default do Normals>Conform and Normals>Set to Face on all geo in Maya.

  2. Is the geo frozen (Translate, Rotate, Scale) and the pivots of all objects are at (0,0,0) in worldspace?

    This ensures all Alembics will be at the right position under the world later on and its also cleaner when rigging. In Maya you can run this:

    import pymel.core as pm
    
    selected_nodes = pm.ls(sl = True, fl = True)
    
    for selected_node in selected_nodes:
    
        #freeze
        pm.makeIdentity(selected_node, a = True)
        #reset pivot
        pm.xform(selected_node, piv = (0,0,0))
    
  3. Doublesided and Opposite turned off?

    This ensures that all faces point to the desired direction for all renderers without custom correction. In Maya you can do this in the Attribute Spreadsheet under the Render tab. (The first two tabs)

    Double Sided and Opposite attributes on the shape
  4. No empty UV Sets floating around and all primary UV Sets are enabled?

    In case there are empty UV sets it makes sense to delete them. Also for some reason Maya likes to have its primary UV set named map1. While this is not always necessary, its good to follow this convention if possible.

    Primary UV set called map1
  5. No overlapping UVs on all the primary UV sets?

    Maybe sometimes there might be reasons to do otherwise but in general its a good idea to have UV sets not overlapping, so that follicles or other rigging nodes that are placed on the surface based on UVs work correct.

  6. UVs have the correct winding order? (Are the UV shells blue instead of red?)

    This happens through flipping and freezing of geometry for example. Can be fixed with Edit UVs>Flip

    UV winding order
  7. Do objects have the right scale?

    Default units in Maya are cm.

  8. Only geo nodes in scene?

    No history, random materials (for example from .obj import) or other nodes should pollute the scene. All geo should have the default Lambert material assigned.

  9. Geometry usefully grouped?
    grp_geo
    grp_body

    grp_left_leg...

  10. All geometry that is actively simulated (cloth etc.) needs to be all quads and tris!

    Collision geometry is fine with NGons. When in doubt ask Johannes.

Animation checklist

Note

Here are some important things for the animators amongst us. Since your output is used in simulation, lighting, shading etc. please pay attention to these points, to ensure everything is running smoothly through the pipe.

If you have questions just ask Johannes, Manuel or Timm.


Simulation Preroll

In order for the extensive cloth sim, that we want to put on top of the helga characters, there is some pre-roll time necessary. Trust me, it’s not rocket science, here’s how we wanna do it.

There are 2 steps that come before the character animation that are needed for the sim dudes.

  1. Frame: 901 - 951

    Character is in T-Pose. The cloth settles on the T-Pose character.

  2. Frame: 951 - 1001

    On frame 951 there is a keyframe of the T-Pose. From here the character morphs linearly into the start pose. Here the cloth gets ready for the real action.

  3. Frame: 1001

    This is the first keyframe of the character animation.

We choose frame 1001 as the first keyframe of animation because it’s a round 1000 frames offset from the usual frame 1.

Photogrammetry checklist

Note

Here i note the settings which seem to provide a good quality/time ratio. (For final solves). They are based on the first days of testing and finding out what works/doesnt work.

If you have questions ask Johannes or Timm.


Solve settings (final)

Set the settings in the batch process dialog to the following. Before the aligning of chunks (second step) it makes sense to check the solved chunks and disable those that dont work before aligning and merging chunks.

Note

  1. Import as multiple chunks

  2. Align Photos: High

  3. Align Chunks:
    • chunks count < 10: High
    • chunks count > 10: Medium
  4. Build dense cloud:
    • chunks count < 10: High
    • chunks count > 10: Medium
  5. Mesh:
    • Face count: Custom (2.000.000 polies).
  6. Texture:
    • Size: 8192²
    • Format: .jpg

Folders to chunks

How folders in the prop_x/photoscan/photos dir. map to chunks in photoscan.

Photogrammetry assets folders and import

Machine list

The following machines have a Photoscan license:

  • Pegasus (Manuel)
  • Gnocchi (Timm)
  • Maccheroni (Timm)
  • Adonis (Johannes)
  • Fusilli (Silke)
  • Gemelli (Nicole)

Asset checklist

Note

In order for the smooth, customized export of assets from animation to lighting/shading, sim and everybody else, the assets need a little treatment before going in production. Here’s a little list of things to take care of. It is advisable (although not needed) to use the AssetManager to automate a lot of these steps in Maya.


  1. Asset has blank prop or char metadata node ?

    This tells the asset manager which categories you want to export. The node options are left blank, because they need to be filled in the scene as a local reference edit.

    Prop metadata node

    (Shot metadata nodes are embedded in the shot files, and never belong into an asset).

  2. Export Attributes attached (proxy, rendergeo, locator...)?

    This tells the asset manager what the content of the categories is. Do all the export objects have the correct attributes assigned. The following attributes are typical:

    1. helga_proxy
    2. helga_rendergeo
    3. helga_locator
    4. helga_highpoly_rendergeo <– Points to the highpoly render version of the asset.

    These attributes are always assigned on the transform nodes of the objects in Maya. No attributes are put on shape nodes by default. (Although its perfectly fine by artists to do so).

    The asset manager tool provides an interface to add or remove those attributes on selected nodes:

    Add or remove attributes with the asset manager.

    Only objects with export attributes will be considered for the export

  3. helga_highpoly_rendergeo attribute on locator set?

    The helga_highpoly_rendergeo attr. is filled out on the asset before referencing. It points to the highpoly render replacement geometry, usually an alembic file located in cache/alembic/highpoly_rendergeo.

  4. Does the asset have a material?

    Make sure the textures are lightweight and published in the textures/props/prop_xy directory.

  5. Does the asset have a rig, and are the joints and locator hidden?

  6. Did you check the asset with the AssetManager after publishing?

    Always be the first to reference in your newly published asset and check if it performs with the AssetManager.

    Check a prop after publishing to see if it performs with AssetManager.

I am a

I am a ...

Here you can quickly find the important ins & outs sorted for your specific task.

I am a texture painter

Note

Hi Marco, you radical machine!
Here is a quick summary of the important things for you...

Where to put your Mari, Mudbox projects or PS files?
Please work on your textures in this folder
Y:/Production/3d/maya/scenes/assets/work/chars/CHARACTERNAME/textures
Y:/Production/3d/maya/scenes/assets/work/props/PROPNAME/textures

Please version up like this:
ulfbert_textures_a_0001_mh.mra
ulfbert_textures_a_0002_mh.mra
ulfbert_textures_b_0003_mh.mra
...
chimney_textures_e_0005_mh.mud
chimney_textures_f_0006_mh.mud

Where to put the textures for the shading artist to use?
Please put the textures you want Manuel (or me) to use in
Y:/Production/3d/maya/scenes/assets/textures/CHARACTERNAME

Please name them without version (this is called publishing in fancy TD speech):
bump.1001.tga
bump.1002.tga
...
color.1001.tga
color.1002.tga
...
etc.

It is just important, that you overwrite your older files and dont version here, so that the
textures that are used in the shaders update themselves.

What formats for textures?
Simply put: Everything is .tga except displacement maps which are .exr
and maps from photoscan which are .jpg

Should you have any questions, you are totaly welcome to email Manuel or me ;)

I am a rigger

Rigging artist information

I am a modeler

Modeler information

I am a pipeline programmer

Please look at this and that .

I am a concept artist

Concept Artist information

I am a animator header

I am a animator

Hey animation crew, here is a quick overview of the what, where and how within the *helga* animation pipeline. Dont be scared, we try to make life for you as easy as possible! Please risk a quick look at these points, and in a spin you should be ready to animate :D


Who can help if there are problems?
Direction: Marco
Animation: Cindy
Rigs: Hanna and Arash
Pipeline: Timm

Dont be afraid to ask as often as possible! Remember: Some TDs look more grim than they really are ;D


Where do i get those rigs?
You can find them here:
Y:/Production/3d/maya/scenes/assets/chars

Important: If the rigs are not there or do not behave as expected please
dont fix this yourself or grab them from another location.
Instead please ask our riggers Hanna and
Arash, since its their responsibility
to supply you with correct working material (and also they are sick experts in the biz of rigging...much more than u and me)

Where are the shots?
Y:/Production/3d/maya/scenes/shots

Why is everything an asset and not just a model or rig?
Assets are nothing special, they are just models and rigs that meet special
criteria. An asset has metadata for example to guarantee that it will
export correctly for the lighting dudes.
The general idea is, that everything you animate (Set, Characters and Props)
is an asset so a smooth functioning is guaranteed.

I want an additional asset in my shot. How?
You need another beer bottle or an empty glass to get the perfect framing?
Its totally fine to reference another asset in your shot, as long as you
grab them from here:
Y:/Production/3d/maya/scenes/assets/props

Those assets have been messed with by a TD and are guaranteed to export
correctly to lighting, simulation etc.
If the asset you need isnt there, just ask me and i will try to add it.

Preroll?
After animation is approved, the simulation department (Johannes)
will do its thing.
In order for them to do a good job, you need to supply them with a so called
Resting Pose. The resting pose basically is a T-Pose that blends linear into
your first frame of animation. You can read more about it here.

Pipeline

API Overview

Foreword

First let me say that API is probably a little misleading since the helga package is mostly a collection of custom tools written in Python, which are independent of each other (like helga.nuke.reconstruction.renderReconstructVRay). But there might be modules in the future that are ment to be exclusively building blocks for programmers.


Organization

Our pipeline is split up into several kinds of packages that import from a logical hierarchy. We can add packages at any depth in the hierarchy at any time.

Category packages

Please note the packages named after contributors. Packages without a name are assumed to be completely written by me (Timm Wagener). If you want to contribute just ask and you will get your own category package.

Warning

Please never contribute outside of your own category package!


Work and Deploy

There are two Helga packages located under:

  • production/scripts/work:

    Here is where we work and edit our scripts. You are expected to always work here and copy your stuff over to deploy when ready. If you work and edit in deploy to save the copy step, you will sooner or later loose work, since i will often delete and recopy the whole helga package.

  • production/scripts/deploy:

    Here is where the DCCs use our scripts and their compilers do the byte compilation (create .pyc files)

This is basically source control for the poor man, but since the helga package was initially ment to be exclusively my own sandbox we have to deal with that inconvenience.

Coding Convention

If you want to contribute to our pipeline, please let me know. There is a place for these efforts and i will try to support you with that. In return you are expected to follow our coding standards in order for your code to integrate consistently with the codebase and make it easier for everybody.

Our coding convention in one sentence:

Do it as in PEP 8 or PEP 257 and name your objects as in the Google Python Convention.

Warning

The current codebase contains large portions of legacy code. While still functional it was created before the coding conventions where active. This will be refactored over time.


Important PEPs

There are a lot of PEPs. Some are more important than others. You are expected to read and use them. Often times in this document, i will just refer to the appropiate place in some PEP.


Variables, Classes, Modules, Packages etc.

For naming of objects in Python we use the Google naming convention. (PEP 8 does not recommend any naming).

module_name, package_name, ClassName, method_name, ExceptionName, function_name,
GLOBAL_CONSTANT_NAME, global_var_name, instance_var_name, function_parameter_name,
local_var_name.

You can find the full documentation here.


Tabs or spaces

PEP 8

Different operating system header

Running on a different OS

Note

The helga pipeline is developed/maintained foremost on the windows platform. However there might be the need sometimes to switch to a different operating system like Linux. For these cases you can find a quick guide here how to start the DCCs in the correct environment, but keep in mind, that support for operating systems other than windows at the moment is very experimental.

Warning

This guide and method are considered temporary and might become deprecated once there is time for a refactoring of the HelgaLauncher tool that is used under windows.


Tested on

  • CentOS7
    • Houdini: Yes
    • Nuke: No
    • Maya: Untested

Step by step guide

Please note that this guide is geared towards TDs or people that are a little familiar with git or Python, since there is. a little effort involved in making it run outside of how it is ment to be run. 6 little steps to take on your own risk :D

  1. Temporary clone the helga git repo

    Clone the helga repo to a temporary location. Just navigate to some temp folder and execute this snippet in your git enabled shell:

    git clone https://github.com/timmwagener/helga.git helga
    

    This should create a folder named helga at your current path and clone the helga repository in it.

  2. Helga folder structure

    Now in your temporary helga repo navigate to helga/general/setup/direct_access/supplementary_material/folder_structure. There you will find a zip file called helga.zip, that contains the helga folder structure. Extract this zip to where you want the helga pipeline and project to be located. PS: The helga folder structure is based on the Maya project structure which might be unfamiliar to non Maya users.

  3. The real clone - work repository

    Now its time to do the real clone of the helga repo and fit it into the new folder structure. Navigate to your_root/Production/scripts/work and execute

    git clone https://github.com/timmwagener/helga.git helga
    

    there again. This is the repo you will keep up-to-date and maintain and always copy into sandbox or deploy when you want to publish changes to production.

  4. Start procedure for Houdini, Maya & Nuke

    Now its time to adjust the startup module for your dccs (Supported are Maya 2014, Nuke 8.x and Houdini 13.0.xxx). Please go to Production/scripts/sandbox/helga/helga/general/setup/direct_access/yaml where you will find a file called pipeline_base_data.yaml. Open it with a text editor, for example sublime text or notepad and adjust the following values:

    1. PIPELINE_BASE_PATH: “path/to/where/Production/and/Organization/folders/are/under”
    2. PIPELINE_FLAVOUR: “deploy” <– Pick either sandbox or deploy. This is the folder name where the script environment you want to use is under.
    3. MAYA_EXE: “path/to/maya.exe”
    4. NUKE_EXE: “path/to/Nuke8.0.exe”
    5. HOUDINI_EXE: “path/to/houdinifx.exe” (or houdini.exe)

    Save and you are good.

  5. Pipeline repo production copies - sandbox and deploy

    Those two fellas could also be named experimental and stable, beta or gold......you get the idea. Just go and copy the whole helga folder, in which you just did modifications, and __init__.py file from work into sandbox and deploy.

  6. Finally

    Test if the application runs correctly. Navigate to deploy|sandbox/helga/helga/general/setup/direct_access and type the following in a python enabled shell:

    #Houdini:
    python -c 'import direct_access;reload(direct_access);direct_access.run("houdini")'
    #Maya:
    python -c 'import direct_access;reload(direct_access);direct_access.run("maya")'
    #Nuke:
    python -c 'import direct_access;reload(direct_access);direct_access.run("nuke")'
    

    If everything went well, your DCC should start within the helga pipeline environment (either sandbox or deploy)


Hopefully that was not too confusing ;) Please remember that you now have the pathes and setup in place, the structure if you will, but you still need a lot of content like .otls for example. In your own production you would of course populate the pipeline with your own scripts, plugins, digital assets, gizmos etc.

Good luck :D

API Documentation

helga.houdini helga.houdini package
helga.maya helga.maya package
helga.nuke helga.nuke package
helga.general helga.general package

Indices and tables