Welcome to CSjark’s documentation!¶

Wireshark and dissectors¶

This section gives a brief introduction to Wireshark and dissectors. The rst part describes what Wireshark is and what it can be used for. The second part explains exactly what a dissector is, and how a dissector can be used to extend Wireshark.

Wireshark¶

Wireshark is a program used to analyze network traffic. A common usage scenario is when a person wants to troubleshoot network problems or look at the internal workings of a network protocol. An important feature of Wireshark is the ability to capture and display a live stream of packets sent through the network. A user could, for example, see exactly what happens when he opens up a website. Wireshark will then display all the messages sent between his computer and the web server. It is also possible to filter and search on given packet attributes, which facilitates the debugging process.

In Figure 1, you can see a view of Wireshark. This specific example shows a capture file with four messages, or packets, sent between internal 2 processes, in other words it is a view of messages sent by inter-process communication. Each of the packets contain one C struct. To be able to display the contents of the C struct, Wireshark has to be extended. This can be accomplished by writing a dissector for the C struct.

Figure 1: Wireshark

/*
* Sample header file for testing Lua C structs script
* Copyright 2011 , Stig Bjorlykke <stig@bjorlykke.org>
*/

#include <time.h>

#define STRING_LEN 30

struct internal_snd {
    int type;
    char name [STRING_LEN];
    time_t time;
};

From struct definition to Lua dissector¶

This section explains what happens under the hood of a Lua dissector.

--
--  A sample dissector for testing Lua C structs scripts
--  Copyright 2011, Stig Bjorlykke <stig@bjorlykke.org>
--

local PROTOCOL = Proto ("internal_snd", "struct internal_snd")
local luastructs_dt = DissectorTable.get ("luastructs.message")

local types = { [0] = "None", [1] = "Regular", [42] = "Secure" }

local f = PROTOCOL.fields
f.type = ProtoField.uint32 ("internal_snd.type", "type", nil, types)
f.time = ProtoField.absolute_time ("internal_snd.time", "time")
f.name = ProtoField.string ("internal_snd.name", "name")

function PROTOCOL.dissector (buffer, pinfo, tree)
   local subtree = tree:add (PROTOCOL, buffer())
   pinfo.cols.info:append (" (" .. PROTOCOL.description .. ")")

   subtree:add (f.type, buffer(0,4))
   subtree:add (f.name, buffer(4,30))
   subtree:add (f.time, buffer(34,4))
end

luastructs_dt:add (1, PROTOCOL)

Dependencies¶

CSjark is written in Python 3.2, and therefore needs Python 3.2 (or later) to run. Latest implementation of Python can be downloaded from Python website. For installing please follow the instruction found there.

There are 4 third party dependencies to get CSjark working:

1. PLY (Python Lex-Yacc)

   PLY is an implementation of lex and yacc parsing tools for Python. It is required by pycparser. Instructions and further information can be found on the page linked above.

   Required version	3.4
   Download location	http://www.dabeaz.com/ply/

2. pycpaser

   Pycparser is a C parser (and AST generator) implemented in Python. Due to the continuous development, CSjark requires the latest development version (not the release version).

   Required version	latest development version from pycparser repository
   Download location	pycparser repository: http://code.google.com/p/pycparser/source/checkout

3. C preprocessor

   CSjark requires a C-preprocessor. The way how to get one depends on operating system used by the user:

   Windows	Bundled with CSjark.
   OS X, Linux, Solaris	Needs to be installed separately. For example, as a part of GCC

4. pyYAML

   pyYAML is a YAML parser and emitter for the Python programming language. YAML is a standard used to specify configurations to CSjark. The website includes both a way to download the software and also instructions of how to install it.

   Required version	3.10
   Download location	http://pyyaml.org/wiki/PyYAML

Wireshark¶

Wireshark is an open source protocol analyzer which can use the Lua dissectors generated by CSjark. To get the proper integration of Lua dissectors, the latest development version of Wireshark is required.

CSjark¶

CSjark can be obtained at git CSjark repository: https://github.com/eventh/kpro9/. CSjark itself requires no installation. After the steps described in the dependencies section is completed. It can be ran by opening a terminal, navigating to the directory containing cshark.py and invoking as described in section Using CSjark.

Configuration file format and structure¶

Format

The configuration files are written in YAML which is a data serialization format designed to be easy to read and write. The configuration must be put in a .yml file and specified when running CSjark as a command line argument (more about CLI in section Using CSjark).

Basic YAML syntax is shown on following example:

# comments can start anywhere with number sign (#) and continues until the end of the line

key1: value1                   # associative array of two key-value pairs
key2: value2                   # pairs are separated by colon (:) and space ( ) on a separate line

{key3: value3, key4: value4}   # inline format of specifying associative arrays
                               # pairs are separated by comma (,) and enclosed into curly braces ({})

- item1                        # list of two items
- item2                        # each item starts with hyphen (-) and space ( ) on a separate line

[item3, item4]                 # inline format of the list
                               # items are separated by comma (,) and enclosed into square brackets ([])

Data structure hierarchy in YAML is maintained by outline indentation (whitespace is used, tab not allowed). All the basic elements can be combined to create a hierarchy:

Options:
    use_cpp:                True
    generate_placeholders:  True

Structs:
  - name:   struct1
    id:     [10, 12, 14]
  - name:   struct2
    id:     [11, 13, 15]

Strings are ordinarily unquoted, but may be enclosed in double-quotes (”), or single-quotes (‘). The specific number of spaces in the indentation is unimportant as long as parallel elements have the same left justification and the hierarchically nested elements are indented further. This sample defines an associative array with 2 top level keys: one of the keys, “Structs”, contains a 2 element array (or “list”), each element of which is itself an associative array with differing keys.

Detailed specification of YAML syntax can be found at YAML website.

Structure

CSjark configuration files might consist of two main parts:

• The first part is used for specifying all the configuration corresponding CSjark processing in general. More about CSjark options in Options Configuration.
• The second part contains configuration for individual C struct definitions. That is described in section Struct Configuration.

The configuration file may have following structure:

Options:
  # there will be all your CSjark processing configuration
  use_cpp: True
  ...

Structs:
  # there will be a sequence of Struct definition configurations
  - name: struct1
    id: [10, 12, 14]
    # another struct1 config
  - name: struct2
    id: [11, 13, 15]
    # another struct2 config

Automatic generation of configuration files

Automatic generation of configuration file is a simple feature, that could save the user of the utility some time, since the essential part of the configuration file is generated automatically. The utility will only create a new file containing the name of the struct and line to specify the ID for the dissector. To generate the configuration file, the utility must be run with -p or --placeholders as an option (see Using CSjark for more about CSjark CLI).

Struct Configuration¶

Each individual C struct processed by CSjark can be treated in different way. All the configuration settings must be done in the Structs section of the configuration file. Every Struct definition is one item of the sequence and may contain these attributes:

General notes

• Definition of Structs part of configuration is not mandatory. However, the user must be aware that if a struct configuration is not defined (namely the id attribute), it can be dissected only as a part of other struct (as its struct member). Otherwise there will be no dissectors registered for the struct.

• If there exists a configuration for a struct member and also configuration for the type of this member, the behaviour is not defined. It is up to the user to ensure the definitions are exclusive for each struct member. For example, in the Value ranges section example, if the percent is defined as float, the configuration would be ambiguous and there would be no guarantee that percent value is between 0 to 100 or -10 to 10.

• If a struct contains another struct as its member, none of the configuration valid for the outer struct is applied on the nested struct. The same goes for unions. In order to configure the nested struct, the user must define separate struct configuration for it. In this example, the configuration valid for the members of person struct is not valid for members of address struct

  struct address {
      int housenum;
      string street;
  };
  
  struct person {
      string name;
      address adr;
      int age;
  };
  

Structs:
  - name: "Name of the struct"
    id: 989
    ranges:
        - member | type: "Name of struct member / type"
          min: "Lowest allowed value"
          max: "Highest allowed value"

Structs:
  - name: example_struct
    id: 90
    ranges:
        - member: percent
          min: 0
          max: 100
        - type: float
          min: -10.0
          max: 10.0

- member | type: member name | type name
  values: [value1, value2, ...] | { key1: value1, key2: value2, ...}
  strict: True | False

Structs:
  - name: enum_example1
    id: 10
    description: Enum config example
    enums:
      - member: weekday
        values: {1: MONDAY, 2: TUESDAY, 3: WEDNESDAY, 4: THURSDAY, 5: FRIDAY, 6: SATURDAY, 7: SUNDAY}
      - type: int
        values: [Black, Red, Blue, Green, Yellow, White]
        strict: True # Disable warning if not a valid value

Structs:
  - name: example
    id: 1000
    description: An example
    bitstrings:
      - member: flags
        0: In use
        1: [Endian, Big, Little]
        2-3: [Platform, Win, Linux, Mac, Solaris]
      - type: short
        0: Red
        1: Green
        2: Blue

Structs:
    - name: structname
      id: 10

Structs:
    - name: structname
      id: [12, 43, 3498]

# CSjark configuration file

Structs:
    - name: custom_lua
      cnf: etc/custom_lua.cnf
      id: 1
      description: example of external custom Lua file definition

#.COMMENT
    This is a .cnf file comment section
#.END

#.DEF_HEADER super
-- This code will be added above the 'super' field definition
#.END

#.COMMENT
    DEF_BODY replaces code inside the dissector function.
    Use %(DEFAULT_BODY)s or {DEFAULT_BODY} to use generated code.
#.DEF_BODY hyper
-- This is above 'hyper' definition
%(DEFAULT_BODY)s
-- This is below 'hyper'
#.END

#.DEF_FOOTER name
-- This is below 'name' definition
#.END

-- This text would be discarded.

#.DEF_EXTRA
-- This was all the Field definitions
#.END


#.FUNC_HEADER precise
    -- This is above 'precise' inside the dissector function.
#.END


#.COMMENT
    FUNC_BODY replaces code inside the dissector function.
    Use %(DEFAULT_BODY)s or {DEFAULT_BODY} to use generated code.
#.FUNC_BODY name
    --[[ This comments out the 'name' code
    {DEFAULT_BODY}
    ]]--
#.END

#.FUNC_FOOTER super
    -- This is below 'super' inside dissector function
#.END

#.FUNC_EXTRA
    -- This is the last line of the dissector function
#.END_OF_CNF

struct custom_lua {
    short normal;
    int super;
    long long hyper;

    char name;
    double precise;

};

-- Dissector for win32.custom_lua: custom_lua (Win32)
local proto_custom_lua = Proto("win32.custom_lua", "custom_lua (Win32)")

-- ProtoField definitions for: custom_lua
local f = proto_custom_lua.fields
f.normal = ProtoField.int16("custom_lua.normal", "normal")
-- This code will be added above the 'super' field definition
f.super = ProtoField.int32("custom_lua.super", "super")
-- This is above 'hyper' definition
f.hyper = ProtoField.int64("custom_lua.hyper", "hyper")
-- This is below 'hyper'
f.name = ProtoField.string("custom_lua.name", "name")
-- This is below 'name' definition
f.precise = ProtoField.double("custom_lua.precise", "precise")
-- This was all the field definitions

-- Dissector function for: custom_lua
function proto_custom_lua.dissector(buffer, pinfo, tree)
    local subtree = tree:add_le(proto_custom_lua, buffer())
    if pinfo.private.caller_def_name then
        subtree:set_text(pinfo.private.caller_def_name .. ": " .. proto_custom_lua.description)
        pinfo.private.caller_def_name = nil
    else
        pinfo.cols.info:append(" (" .. proto_custom_lua.description .. ")")
    end

    subtree:add_le(f.normal, buffer(0, 2))
    subtree:add_le(f.super, buffer(4, 4))
    -- This is below 'super' inside dissector function
    subtree:add_le(f.hyper, buffer(8, 8))
    --[[ This comments out the 'name' code
        subtree:add_le(f.name, buffer(16, 1))
    ]]--
    -- This is above 'precise' inside the dissector function.
    subtree:add_le(f.precise, buffer(24, 8))
    -- This is the last line of the dissector function
end

delegator_register_proto(proto_custom_lua, "Win32", "custom_lua", 1)

#.FUNC_FOOTER pointer
    -- Offset: {OFFSET}
    -- Field value stored in lua variable: {VALUE}
#.END

local field_value_var = subtree:add(f.pointer, buffer(56,4))
    -- Offset: 56
    -- Field value stored in lua variable: field_value_var

trailers:
  - name: proto1
    member: trailer_count
    size: 32

trailers:
  - name: proto1
    count: 2
    size: 32

trailers:
  - name: ber
  - count: 4
  - size: 6

Structs:
  - name: custom_type_handling
    id: 1
    customs:
      - type: time_t
        field: absolute_time
      - member: day
        field: uint32
        abbr: day.name
        name: Weekday name
        base: base.DEC
        values: { 0: Monday, 1: Tuesday, 2: Wednesday, 3: Thursday, 4: Friday}
        mask: nil
        desc: This day you will work a lot!!

#include <time.h>

struct custom_type_handling {
    time_t abs;
    int day;
};

Structs:
    - name: unknown struct
      id: 111
      size: 78

Options Configuration¶

CSjark processing behaviour can be set up in various ways. Besides letting the user to specify how the CSjark should work by the command line arguments (see section Using CSjark), it is also possible to define the options as a part of the configuration file(s).

The last 5 options can be also specified separately for each individual input C header file. This can be achieved by adding sequence files with mandatory attribute name.

Below you can see an example of such Options section:

Options:
    verbose: True
    debug: False
    strict: False
    output_dir: ../out
    output_file: output.log
    generate_placeholders: False
    use_cpp: True
    cpp_path: ../utils/cpp.exe
    excludes: [examples, test]
    platforms: [default, Win32, Win64, Solaris-sparc, Linux-x86]
    include_dirs: [../more_includes]
    includes: [foo.h, bar.h]
    defines: [CONFIG_DEFINED=3, REMOVE=1]
    undefines: [REMOVE]
    arguments: [-D ARR=2]
    files:
      - name: a.h
        includes: [b.h, c.h]
        define: [MY_DEFINE]

Platform specific configuration¶

To ensure that CSjark is usable as much as possible, platform specific

Entire platform setup is done via Python code, specifically platform.py. This file contains following sections:

1. Platform class definition including it’s methods
2. Default mapping of C type and their Wireshark field type
3. Default C type size in bytes
4. Default alignment size in bytes
5. Custom C type sizes for every platform which differ from default
6. Custom alignment sizes for every platform which differ from default
7. Platform-specific C preprocessor macros
8. Platform registration method and calling for each platform

When defining new platform, following steps should be done. Referenced sections apply to platform.py sections listed above. All the new dictionary variables should have proper syntax of Python dictionary:

Field sizes

Define custom C type sizes in section 5. Create new dictionary with name in capital letters. Only those different from default (section 3) must be defined.

NEW_PLATFORM_C_SIZE_MAP = {
    'unsigned long': 8,
    'unsigned long int': 8,
    'long double': 16
}

Memory alignment

Define custom memory alignment sizes in section 6. Create new dictionary with name in capital letters. Only those different from default (section 4) must be defined.

NEW_PLATFORM_C_ALIGNMENT_MAP = {
    'unsigned long': 8,
    'unsigned long int': 8,
    'long double': 16
}

Macros

Define dictionary of platform specific macros in section 7. These macros then can be used within C header files to define platform specific struct members etc. E.g.:

#if _WIN32
    float num;
#elif __sparc
    long double num;
#else
    double num;

Example of such macros:

NEW_PLATFORM_MACROS = {
    '__new_platform__': 1, '__new_platform': 1
}

Register platform

In last section (8), the new platform must be registered. Basically, it means calling the constructor of Platform class. That has following parameters:

Platform(name, flag, endian, macros=None, sizes=None, alignment=None)

where

name	name of the platform
flag	unique integer value representing this platform
endian	either Platform.big or Platform.little
macros	C preprocessor platform-specific macros like _WIN32
sizes	dictionary which maps C types to their size in bytes

Registering of the platform then might look as follows:

# New platform
Platform('New-platform', 8, Platform.little,
         macros=NEW_PLATFORM_MACROS,
         sizes=NEW_PLATFORM_C_SIZE_MAP,
         alignment=NEW_PLATFORM_C_ALIGNMENT_MAP)

Coding Style Standard¶

The programming language used to implement the utility is Python. All the code should as much as possible follow the coding style described in the Style Guide for Python Code (PEP8). In addition we decided that the design should attempt to be pythonic, as detailed by PEP20.

Code base¶

The entire CSjark project is hosted on GitHub. In order to get the source code, you can:

• clone the Git repository: https://github.com/eventh/kpro9
• download most recent tar archive
• download most recent zip archive

Issue tracking¶

For issue tracking, we use bug tracking capabilities of GitHub. You can register bugs, discuss issues or see what’s going on for the next milestone, both from an email and from a web interface.

https://github.com/eventh/kpro9/issues

Textual description¶

CSjark starts with the csjark module. It takes as input command line arguments, including file and folder names for C header files and configuration files. It replaces folder names with the file names of the files inside the folders, and checks that all the files exists, then it gives the names of the configuration files to the config module.

The config module parses configuration files and stores them in suitable config data structures in memory. It takes as input file names, and it opens those files to read their content.

Csjark module then gives file names of C header files to the cpp module. The cpp module gives the file names and some other arguments to an external program, the C preprocessor (cpp.exe on Windows). This external program opens the header files, and when it encounters #include directives it searches for the right file and opens it as well. The output of this external program is just a long string of C code, which cpp module returns to the csjark module.

The csjark module then gives the C code string to cparser module, which forwards the string to pycparser. Pycparser parse the C code to generate an abstract syntax tree, which it returns to cparser which returns it to csjark.

Csjark module then tells cparser to find all struct definitions in the abstract syntax tree, which it does by traversing the tree. Each time it finds a struct, it asks dissector module to create a protocol for it. Afterwards cparser holds a list of all the created protocols.

Csjark then gets the list of protocols from cparser, and for each one asks dissector module to generate lua code for Wireshark dissectors. It writes these dissectors to files, and finishes with a status message informing the user how it all went.

The new field module is simply to move class Fields and its sub-classes into their own module to make dissector module smaller and less complex.

Summary

csjark module writes Lua files, config module opens and reads YAML files, cpp module starts an external program which reads C header files. The structure as well as the associations among the classes are shown on following module and class diagrams.

Module diagram¶

CSjark: module diagram

Class diagram¶

CSjark: class diagram

White box testing¶

This type of white box testing basically means verification of functionality of specific section of the code, usually at the function level. As a tool for creating white box unit tests, the team decided to use the Attest testing framework for python code.

Black box testing¶

In general black box testing does not require the tester to have any intimate knowledge about the system or any of the programming logic that went into making it. Black box test cases are built around the specifications and requirements of a system, for example its functional, and in some cases, non-functional requirements.

For CSjark, black box testing means feeding the utility with input C header file, corresponding configuration file and generating the output (Lua dissector). Then, the generated output is compared to expected output.

Another way how to do the black box testing to use the generated dissectors in Wireshark. For this purpose, we also need a pcap file containing the IPC packets that corresponds to the input C header files. You can read more about whole process in the User Manual section Introduction. A short guide how to create pcap file for the C struct can be found in the project wiki.

With the generated Lua dissectors, it should possible to display the contents of the C struct within the IPC packets. Also, according to the utility requirements, it should be possible to filter and search by any variable name or its value. This way of testing is based on manual checking of the individual variable values. The process involves several manual steps and therefore cannot be automated.

Regression testing¶

The test must be written in a way that it should be possible to run them repeatedly after the first run. That can ensure that all the implemented functionality is still working well after changes in the code.

Creating tests¶

To create tests using Attest, you start by importing Tests, assert_hook and optionally contexts from attest library. You then create a variable and initialize it to an instance of Tests, which is the variable that will contain list functions that each constitutes one test that is to be run. To feed your test instance with functions for testing you then have to mark these functions with a decorator and feed it the .tests function of the Tests instance. After creating a unit test in this fashion you can run all of your unit tests through Attest from the command line by typing:

python -m attest

This runs all of your unit tests through Attest and returns a message telling the user how many assertions failed, as well as what input made them fail. For more information read the user documentation of Attest.

Testing code¶

All the test code, the testing configuration files and the testing input files are located in folder

CSjark/csjark/test

It contains modules for unit/module testing of each of the CSjark modules as well as modules for bundled white/black box testing.

Documentation files in your local checkout¶

Most of the CSjark’s documentation is kept in CSjark/docs. You can simply edit or add ‘.rst’ files which contain ReST-markuped files. Here is a ReST quickstart but you can also just look at the existing documentation and see how things work.

Automatically test documentation changes¶

We automatically check referential integrity and ReST-conformance. In order to run the tests you need sphinx installed. Then go to the local checkout of the documentation directory and run the Makefile:

cd CSjark/docs
make html

If you see no failures chances are high that your modifications at least don’t produce ReST-errors or wrong local references. Now you will have .html files in the _build documentation directory which you can point your browser to!

Additionally, if you also want to check for remote references inside the documentation issue:

make linkcheck

which will check that remote URLs are reachable.

Automatic builds on Read The Docs¶

The CSjark project documentation is hosted at ReadTheDocs. Each commit to the repository triggers a new build of the documentation, therefore it always remains up to date.