Y U NO have automated tests?¶

Python¶

Yuno uses Python, either 3.x or 2.7. If you need it, use your package manager or get a copy and install it somewhere nice.

Not sure if you have Python, or which version? On Linux/OS X, use:

# Check for any version
python -V

# Check for Python 3
python3 -V

On Windows:

:: Any version
python -V

:: Python 3
py -3 -V

Yuno¶

cd /your/project
git clone https://github.com/bulatb/yuno.git
cd yuno
git checkout v0.3

your/project/
    .eclipse_turds, .idea_spam, etc/
    bin/
        (.class files)
    src/
        (.java files)
    testcases/
        (we'll get to this)
    yuno/
        (yuno.py, etc)

Make it nice¶

At this point, Yuno can be run with either

./yuno.py <args>

or

<python executable> yuno.py <args>

A simple yuno <args> would be much nicer. We’ll use a stupid, brute-force alias because it makes things easy, but a symlink, %PATH% and %PATHEXT%, or whatever you prefer should work as well. (A future version will remove this step. For now... sorry!)

On Linux/OS X:

alias yuno='/path/to/compatible/python /path/to/yuno/yuno.py'

On Windows:

:: Replace XX with your Python version - 27 for 2.7, 31 for 3.1, etc.
doskey yuno=C:\PythonXX\python.exe C:\path\to\yuno\yuno.py $*

To make it permanent:

• On Linux/OS X with Bash, add your command to ~/.bashrc or similar.
• On Windows, make a special shortcut for your cmd.exe.

Layout¶

Structure your repository any way you want, as long as answer files are always in the same directory as corresponding tests. By default, Yuno looks for files with .ans.out and .rc extensions in a repo named testcases/ next to yuno/, but those values can be changed in the configuration file.

CSE 131¶

Templates with the recommended layouts are available for project 1 and project 2. These layouts are compatible with older testing tools and work with special Yuno features.

Signatures¶

yuno run all | failed | failing | passed | passing | <glob>
yuno run phase <#> | check <#> | suite <name> | files <glob>
yuno run phase <#> check <#>
<newline-delimited stream> | yuno run -

Options¶

--diff [mode]

Print a diff for any tests that fail. The mode can be context or unified, defaulting to context.

--ignore [regex [regex ...]]

Don’t run tests whose path (including filename) matches any given regex. Python-style patterns; takes (?iLmsux) flags; backrefs can be named or \1, \2, etc.

--pause [event(s)]

Pause on certain events: p (test passed), f (failed), s (skipped), w (warned), or any combination (pf, fsw, etc). Defaults to f.

--save <name>

Save the tests that just ran as a suite called <name>.

-o, --overwrite

Use with save to write over an existing suite with the same name.

Running by folder¶

If run receives one argument and it doesn’t match a special value (all, -, failed, failing, passed, or passing), it’s taken as a Unix path glob representing folders in the repo to be searched recursively. Any files inside that have the right extension (.rc by default) will be run as tests.

Every test in dir1/ or dir2/:

yuno run dir[1-2]

Every test in every check ending in 2:

yuno run phase*/check*2

Every test with a Companion Cube:

yuno run enrichment/chamber17

By phase or check¶

Being a Compilers tool, Yuno gives special treatment to repositories laid out in phases and checks. The phase and check commands can be used separately, together, or not at all—they just provide a nicer wrapper over using globs. Each one’s <#> may be a number, a number and a letter, or dash-separated range. (More on that below.)

Any test files inside matching folders or subfolders will be run.

yuno run phase 1-3
yuno run check 12

# If check alone would be ambiguous
yuno run phase 2 check 6a

To support checks with multiple parts, ranges can slice on numbers and letters together. Each end may be one or more digits, optionally followed by a letter. For example, to run every test in every check between check6b and check10 (inclusive):

# Runs 6b, 6c, 7, ... 9a, ..., 9z, ..., 10z
yuno run check 6b-10

Note: If you don’t put in a range, Yuno looks for an exact match. Asking it to run check 3 means asking it to run tests in a folder called check3, not to run 3a, b, and c together.

If you want to run them all, use:

yuno run check 3a-3c

Or if you’re lazy:

yuno run check 3a-c

Yuno always does its best to run no less than what you asked for, only skipping checks if they’re specifically excluded by the range. A range endpoint without a letter will include that check and all its subparts. Any checks that fall inside the middle of the range are loaded fully, # to #z.

# 4, 4a, 4b, ..., 9, ..., 9d, 9e (but not 9f)
yuno run check 4-9e

# 5b, 5c, ..., 6, 6a (but not 5 or 5a)
yuno run check 5b-6a

# 5, 5a, ..., 10, ..., 10z
yuno run check 5-10

By status¶

Like all, passed/failed and passing/failing can be used to run special sets of tests.

Every test that passed (or failed) on the last run:

yuno run passed
yuno run failed

Every test that hasn’t passed since it last failed:

yuno run failing

Every test that hasn’t failed since it last passed:

yuno run passing

By suite¶

Suites are arbitrary sets of tests, grouped together and named. They’re handy for creating groups of tests that go together without having to move files around.

To run a suite:

yuno run suite <name>

To create a suite, either:

• Use the --save flag with a name (run <whatever> --save <name>), which makes a suite from every test that ran this time; or
• By hand, create <name>.txt in settings/suites/ and add the path for every test you want: one per line, repo-relative, including the file name.

For example, a finished suite accessible as pointers looks like this:

$ cat settings/suites/pointers.txt
phase2/check12/dereference-void.rc
phase3/check18/pass-pointer-by-reference.rc
sizeof/pointer-size.rc

By filename¶

For more precise control over which tests will run, use run files with a glob that matches the full path and name you want.

Only tests from people you trust:

yuno run files public/mallory/*.rc

Tests for printing 5 or fewer lines:

yuno run files **/cout/print[0-5].rc

From a pipe¶

If yuno run - sees text on stdin, it treats it as a newline-separated list of test files and ignores any positional arguments. Options and flags will still be used if they make sense. See the Data section for more on how to use this to hack in some extra capability.

To re-run every test that raised a warning last time:

# Find lines that start with w, clean them up, and pipe to Yuno
$ grep ^w data/last-run.txt | sed 's/^w //' | yuno run -

But no one likes sed, so Yuno knows to strip out its own line labels:

$ grep ^w data/last-run.txt | yuno run -

Setup: Windows¶

Everything is pre-configured. Mount ieng9 so Q:\ maps to /home/linux.../you.

Setup: Linux/OS X¶

On your machine—not ieng9—edit yuno/compile/settings/config.json so the settings match this table. These paths assume you mounted /home/linux/.../you at /mnt/ieng9/.

Running tests¶

yuno compile phase 1 check 2

Getting results¶

The watch you ran will print the normal summary each time it runs a batch. If you’d rather see results on your machine, point "data_folder" in your local settings/config.json at yuno/data on ieng9. Then run yuno show as normal:

yuno show last
yuno show failed
# etc.

Scary warning¶

Yuno is extremely stupid. It has no idea what your tests should do. If any answer files are wrong, it’s possible you won’t find out until you’re getting angry emails from your friends and no one speaks to you in lab. Or worse—until your grades come back and everything was broken.

That’s why this feature is called certify: by running it, you certify your compiler’s output for these cases will be right. Let typing the pretentious name remind you to be careful.

Signatures¶

yuno certify files <glob>
<newline-delimited stream> | yuno certify -

Options¶

--overwrite

If an answer file already exists, overwrite it without asking. Use with caution.

--correct

Don’t ask if output is correct before accepting. Use with terror.

Creating by glob¶

This feature works the same as yuno run files: the <glob> should match specific file names, including full paths and extensions, except the output will be answer files instead of test results. You’ll be prompted every time it tries to overwrite a file unless you use --overwrite.

To generate an answer file for my-first-test.rc:

yuno certify files phase1/check1/my-first-test.rc

Creating by pipe¶

As with yuno run, users with nice shells get extra power here. For example, Unix users can generate answers for any tests that were skipped because of missing answer files:

grep ^s data/last-run.txt | yuno certify -

Options¶

--last-run

Also prune your last run’s log file.

--suites

Also prune the suites in your main suite folder.

--all

Short for --last-run --suites.

Configuration¶

The settings you can change are documented and defined in settings/config.json. Yuno comes pre-configured to work with the standard repo layout described above, but you’re free to use whatever you prefer. The defaults are saved in settings/config.json.default.

Except for comments, lines starting with //, the syntax is standard JSON. Intrepid editors will find that Yuno’s comment stripping code is very stupid, and putting comments at the ends of lines is just a fancy way to break it. Complaints may be addressed to:

Customizing output¶

Most messages that end up at the console (not yet all) are built from plaintext template files whose paths are set in failure_message, diff_message, and so on in the settings file. You can either change the paths or just edit the defaults in-place. What you see is what you get, newlines and all.