Skip to content

Definitions

This directory contains the example definitions which showcase various functionality.

Versioning

Definition use a modified semantic versioning:

  • Patch: adding fields and files to the definition.
  • Minor: modifying existing fields, such as renaming, deleting or changing their meaning. Such changes would either break the definition validation or it would cause unintended/incorrect behavior.
  • Major: no set of rules are defined, can be used for extreme changes to the definition format.

The current definition version the backend supports is the topmost version in the changelog. For example, backend version 2.0.0 supports definitions with version 0.12.1.

Definition file structure

content/
files/
drive/
config.yml
channels.yml
injects.yml
email.yml
milestones.yml
tools.yml
roles.yml
questionnaires.yml
objectives.yml

content/, files/, drive/, and llm/ directories are optional. They do not need to be present if your definition does not use the relevant features.

Furthermore, it is possible to use a directory instead of a single YAML file. The files inside the directory must follow the same structure as the original YAML file. For example, if you decide that injects.yml file is too large you can create a directory injects inside of which you create 4 different files which when combined will contain the original content of the injects.yml file. These files can be named in any way you like, however they must be in the YAML format.

The above currently holds true for all YAML files except config.yml. These approaches can be mixed: e.g. if you have many injects but not that many tools you can put the injects into a directory and still use a single tools.yml file.

Note: If the definition contains both a YAML file and a directory for the same structure, the YAML file will be prioritized and the directory ignored. E.g. if you have injects.yml and injects directory, the directory will be skipped.

Current file formats

Below are schema definitions for each file. Every field is required unless a default value is specified or it is stated otherwise.

Quick list

Common definitions

This section contains definitions of common fields used by multiple definition files. Restrictions on some uses might apply, these restrictions will be listed on the exact location.

Content

Only one of content or content_path can be specified on a single content field. These fields can be formatted as Markdown with these restrictions. The Markdown syntax can be found here.

Warning: The frontend application sanitizes the HTML generated by Markdown using the default options of sanitize-html. Some content may therefore not be shown as intended. Mainly, <script>, <style>, <iframe>, <img>, <video>, and <form> tags, along with attributes like id, class, and event handlers (e.g., onclick, onload) are not allowed. Furthermore, the only URL schemes allowed (in href, src, and cite) are http, https, ftp, mailto, and tel.

If you want to embed media (images, videos, or audio recordings), use files from the definition. To embed a file, simply surround the path to the file in {{}}. For example:

content:
  content: "This is an embedded image {{files/image.png}}"

Warning: If the embedding is done directly in YAML (like the example above), you have to surround the content in ", just like the example.

The same syntax can also be used in markdown files, without the requirement of surrounding the text in ". It is recommended to do embeddings in markdown files instead of specifying it directly in YAML.

All most commonly used image, audio, and video file formats are supported.

  • content: str, default="" – content displayed to users, mutually exclusive with content_path
  • content_path: str, default="" – name of a markdown file that contains the content, mutually exclusive with content
  • attachments: list of strings, default=empty – optional list of file attachments, each file has to be present in the files directory in the definition

Control

  • milestone_condition: str, default="" – condition which must be met before an object can be sent/processed. Supports simple logical expressions in Python format, where the allowed operators are and, or, not and (). These operators can be applied to names of milestones specified in the definition.
  • activate_milestone: list of strings, default=empty – a list of milestone names (not display names), which will be activated after an object is sent/processed
  • deactivate_milestone: list of strings, default=empty – a list of milestone names (not display names), which will be deactivated after an object is sent/processed
  • roles: list of strings, default=empty – usable only if roles are enabled, a list of roles, a team must have at least one of the roles to be eligible for this object

Overlay

  • duration: int, default=0 – number of minutes the overlay should stay active, 0 means it has no time limit, must be greater than or equal to 0

HINT: Because overlay only has a single duration field, specification of an overlay without a set duration is done like so:

# other fields
    overlay: {}

LLM-assessment

  • persona: str, default="" – system prompt for the LLM defining the behaviour and persona of the LLM
  • persona_path: str, default="" – name of a text/markdown file that contains the persona
  • assessment: str, default="" – prompt with specific task context and instructions for assessment and required response format
  • assessment_path: str, default="" – name of a text/markdown file that contains the assessment

Either persona or persona_path must be specified. Either assessment or assessment_path must be specified. If ..._path is used, the file with the given name must be present inside llm/ directory in the definition.

This object allows to specify a persona and assessment for the given:

  • email address
  • question (only free-form type)

The llm_assessment can be created for situations where a longer input is expected to be sent by trainee either:

  • in an email to a ceratin email address or
  • in an answer to a (free-form) question

Clear instructions for the LLM behaviour and role should be provided in persona. The assessment should contain clear instructions for assessment of the trainees input and additional data (what kind of input did the trainee provide, what should be assessed, how, what the output of the LLM should contains,...).

drive

This optional directory can be included in the definition to use the internal drive feature. Files saved in this directory will be available to everyone in the exercise.

config.yml

This file contains various settings for an exercise. All advanced features are disabled by default:

  • name: string, required – Default name when uploading a definition.

  • exercise_type: string, default="" – Type of the exercise.

  • exercise_duration: int – Duration of the exercise in minutes.

  • email_between_teams: bool, default=False – Allow emails between teams. Can only be enabled if emails are enabled.

  • custom_email_suffix: string, default="mail.com" – Changes the domain of email addresses of teams, e.g. if set to gmail.com team email addresses will have the form team_name@gmail.com.

  • show_exercise_time: bool, default=False – Show the remaining duration of the ongoing exercise to trainees on the frontend.

  • show_exercise_overview: bool, default=False – Shows the performance overview to trainees when the exercise is over.

  • enable_roles: bool, default=False – Enables use of team roles. Requires roles.yml file to be included in the definition.

  • instructor_required: bool, default=False – Indicates whether the exercise is intended to be run with an instructor present.

  • description: string, default="" – description of the definition

  • target_audience: string, default="" – target audience of the definition

  • license: string, default="" – License identifier.

  • language: string, default="" – Language of the definition.

  • prerequisites: list of strings, default=empty – list of prerequisites for the definition

  • goals: list of strings, default=empty – List of goals.

  • external_resources: list of strings, default=empty – List of external resources.

  • authors: list of strings, default=empty – List of authors.

  • instructor_notes: content, default=empty – information only available to instructors

  • notes: content, default=empty – General notes about the definition.

  • allowed_file_types: list of strings, default=predefined list of allowed file types - a whitelist of allowed file types specified with their mimetype. By default, the list contains these file types:

    • text/plain
    • text/markdown
    • application/vnd.ms-excel
    • application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
    • application/msword
    • application/vnd.openxmlformats-officedocument.wordprocessingml.document
    • image/png
    • image/jpeg
    • application/pdf

    Note: This list you specify will be the final list. The above defaults will not be added to it. If you wish to allow more file types, you must include the default file types as well. Additionally, you may choose to disallow file uploads entirely during the exercise by specifying an empty list ([]). These restrictions apply to all users for the specific exercise, including instructors and admins.

  • version: string – semver version string of the format this definition uses, see versioning for details.

objectives.yml

This file contains all learning objectives. Each learning objective has the following fields:

  • name: string, unique – name of the learning objective
  • tags: list of strings, default=empty – a list of tags corresponding to this objective
  • description: string, default="" – description of the objective
  • order: int, default=0 – order of the objective, defines the order in which objectives will be displayed for instructors,
  • activities: - list of learning activities
    • name: string – name of the learning activity
    • description: string, default="" – description of the activity
    • tags: list of strings, default=empty – a list of tags corresponding to this activity
    • milestones: list of strings, default=empty – list of milestones related to this activity, a milestone can only be linked to one activity
    • tools: list of strings, default=empty – list of tools related to this activity, a tools can only be linked to one activity
    • injects: list of strings, default=empty – list of injects related to this activity, an inject can only be linked to one activity
    • addresses: list of strings, default=empty – list of addresses related to this activity, an address can only be linked to one activity
    • questionnaires: list of strings, default=empty – list of questionnaires related to this activity, a questionnaire can only be linked to one activity

channels.yml

This file contains the definitions of all channels used in the exercise.

  • name: string, unique – the name that will be displayed to users
  • display_name: string, default=name – display name of the channel
  • description: string, default="" – description of the channel
  • type: string – the type of messages that will be sent to this channel, see injects.yml for details.

Currently supported channel types:

  • info – the most basic type of message, A definition can include one or more channels with the info type.
  • tool – tool usage by trainees will be sent to the channel with this type. Injects cannot be specified with this type and there can be at most one channel with this type in a definition
  • email – injects meant to represent email communication
  • form – questionnaires will be sent to this channel

At most one channel of a specific type can exist (except the info channels). In other words, there cannot be multiple channels with type email. If a definition does not contain a channel with the tool type, no tools can be specified. Same rule applies to emails and questionnaires. When a channel of some type is specified, at least one related object must be specified. E.g. if an email channel exists, at least one email address must be specified in emails.yml.

Note: Defining a channel for tools or emails is required to enable these features. Without such a channel, these functionalities will be turned off.

injects.yml

This file contains a list of injects. Each inject has the following fields:

  • name: string, unique – name of the inject
  • display_name: string, default=name – display name of this inject
  • time: int, default=0 – time since the beginning of the exercise in minutes, the inject will not be processed until this time has been reached
  • delay: int, default=0 – time in minutes, whenever this inject is about to be sent, it is delayed by this number of minutes. The condition is checked throughout the duration and if it becomes false, the inject is cancelled and the process can start again.
  • organization: string, default="" – name of the organization this inject was sent from
  • type: string, default="info" – type of this inject, the alternatives field depends on this value
  • alternatives: options specified bellow
  • target: string, default="" – a mandatory field (only for info injects) that specifies to which info channel should be the inject sent

When selecting alternatives to send, they are ordered alphabetically based on their name. The first one that fulfills the condition is selected. In other words, when multiple alternatives can be sent, their priority is determined by their name attribute.

Alternatives: info

  • name: string – name of the alternative
  • content: content, default=empty - if empty, no ActionLog will be created when this alternative is sent. Can be used for hidden state management of the exercise.
  • control: control, default=empty – milestone conditions are checked when the alternative is being selected, milestone modifications happen after the alternative has been selected
  • overlay: overlay, default=empty
  • confirmation: default=empty – adds a confirmation button to this alternative
    • text: string – the button text
    • control: control – milestone modifications happen after a team presses the button, milestone condition and roles cannot be specified

Alternatives: email

  • name: string – name of the alternative
  • sender: string – an email address from which this alternative is sent, must be an address specified in email.yml.
  • subject: str – subject of the email
  • content: content, default=empty
  • control: control, default=empty – milestone conditions are checked when the alternative is being selected, milestone modifications happen after the alternative has been selected
  • extra_copies: int, default=0 – number of extra copies to be sent, useful for specifying spam email.
  • overlay: overlay, default=empty

tools.yml

This file contains a list of tools. A tool can be thought of as a simple function, which based on its input and current state of the exercise returns some output. Each tool has the following fields:

  • name: string, unique – Name of the tool
  • display_name: string, default=name – display name of this tool
  • category: string, default="" – category of the tool, used to group multiple tools together visually
  • tooltip_description: string, default="" – Description of a tool displayed to trainees.
  • hint: string, default="" – Argument hint
  • button_caption: string, default="Submit" – text displayed on the tool for submitting an input
  • default_response: content, default=Empty – Response that the tool returns if no other response gets matched.
  • roles: list of strings, default=empty – a list of roles which can access this tool. Usable only if roles are enabled. If empty, all roles can access this tool.
  • requires_input: bool, default=True – if set to False, this tool will not require an input and will be showcased as a simple button, will cause the param fields in responses to be ignored when selecting a response
  • responses: - A list of responses. A response is matched when its conditions are met. When trainees use a tool, the list of responses is examined from top to bottom and only the first matched response is sent back.
    • param: string – A parameter that is matched exactly to the input of trainees. Can be written as Python regular expression if parameter regex is set to true. Note: if you need to match a literal backslash (\) in trainee input, you must write 4× as many backslashes in the definition. Example: to match \ write \\\\ in param; to match \\ write \\\\\\\\.
    • regex: bool, default=False – Determines whether parameter param is matched as a string or a regex.
    • time: int, default=0 – Time since the beginning of the exercise in minutes after which this response can be matched. If the trainees use correct param but this time was not reached yet, it will not match.
    • content: content, default=empty
    • control: control, default=empty – milestone condition is checked when a response is being selected, milestone modifications are triggered after this response has been selected

milestones.yml

This file contains all milestones in an exercise. Any milestone_condition or (de)activate_milestone field in the whole definition can only contain milestones defined in here. Each milestone has the following fields:

  • name: string, unique – Name of the milestone. Rules for the naming of the milestones:
    • A name must start with a letter or the underscore character
    • A name cannot start with a number
    • A name can only contain alphanumeric characters and underscores (A-z, 0-9, and _ )
    • Variable names are case-sensitive (age, Age and AGE are three different variables)
    • A name must be continuous string -> a name can not contain spaces
  • display_name: string, default=name – display name of the milestone
  • description: string, default="" – description of the milestone
  • tags: list of strings, default=empty – optional list of tags related to the milestone
  • file_names: list of strings, default=empty – a list of file names included in the files directory. If any of these files is downloaded, this milestone activates.
  • final: bool, default=False – sets this milestone as a final milestone. If a team reaches this milestone, their exercise will be finished. For synchronous exercises, all teams must reach this milestone for the exercise to be finished.
  • initial_state: bool, default=False – the initial state of the milestone, if this is set to True, final cannot be set to True
  • score: int, default=0 – the score for this milestone, will be added to the teams' score while the milestone is reached and subtracted when it becomes unreached; can be negative

email.yml

This file contains all email addresses where each address has the following fields:

  • address: string, unique – Address of the simulated correspondent.
  • description: string – Description of the correspondent and related context for the instructor.
  • control: control, default=empty – milestone condition and roles cannot be specified, milestone modifications are triggered after a team sends an email to this address
  • team_visible: bool, default=False – Specifies if this email address should be visible to the teams.
  • organization: string, default="" – Name of the organization this email address belongs to.
  • signed: bool, default=True – specifies whether this email address will be shown as certified
  • templates: not required – A list of template emails that the instructor can choose from when writing an email.
    • context: string – Description of the context of this email answer for the instructor.
    • subject: string – subject of the email thread when sending this template
    • content: content, default=empty
    • control: control, default=empty – milestone condition and roles cannot be specified, milestone modifications are triggered once the template is sent
  • llm_assessment: llm assessment, default=empty

roles.yml

Required if roles are enabled. This file contains all role names:

  • name: string, unique – Name of the role.
  • display_name: string, default=name – display name of the role
  • description: string, default="" – description of the role

questionnaires.yml

This file contains all questionnaires. Each questionnaire has the following fields:

  • name: string, unique – name of the questionnaire
  • display_name: string, default=name – title of the questionnaire
  • content: content, default=empty – description of the questionnaire
  • time: int, default=0 – time since the beginning of the exercise in minutes, after which the questionnaire will be sent
  • delay: int, default=0 – Time in minutes, how long to wait before sending the questionnaire. The condition is checked throughout the duration and if it becomes false, the timer is cancelled and it will restart when the condition is met again.
  • control: control, default=empty – the milestone condition is checked when the questionnaire should be sent (similar to injects), the milestone modifications are triggered once after the questionnaire has been answered fully (all questions answered)
  • overlay: overlay, default=empty
  • post_exercise_submission: boolean, default=False – allow teams to answer this questionnaire after the exercise is finished, milestone modifications can still occur
  • repeatable: object, default=empty – if specified, teams will be able to submit answers for this questionnaire multiple times until a submission is correct (all answers within the submission are correct) IMPORTANT - milestone modifications from the questions are triggered only if the submission was correct, in case the team reached the maximum number of attempts, the question milestones are ignored and only the control and on_fail milestone modifications are triggered
    • max_attempts: int, default=-1 – maximum number of attempts for this questionnaire, -1 signifies no maximum, after a team reaches this number, the questionnaire will not be answerable again
    • on_fail: control, default=empty – required if max_attempts is not -1, milestone modification that occurs after <max_attempts> number of attempts
  • show_feedback: bool, default=False – after a team answers a questionnaire, they will receive feedback whether their answers were correct, partially correct or incorrect. This feedback does not contain information about the correctness of specific answers and it does not reveal the correct answers. The feedback only pertains to questions which can be automatically evaluated, which means that free-form questions are ignored during evaluation.
  • questions: details specified bellow
    • content: content, default=empty – should contain the text of the question
    • type: string, default="radio" – type of the question, determines what fields can be specified, currently supported values are radio, multiple-choice, free-form and auto-free-form
    • note: string, default="" – add a stylized note after the question

Question: radio

  • max: int, required – the number of options for the question
  • labels: list of strings, default=empty – a list of labels to display as options, empty list means the labels will be numbers in range [1-max], if specified, the number of labels must be equal to max
  • correct: int, default=0 – position which represents the correct choice, for example, if we have 4 labels yes, no, maybe, absolutely, the correct field should have the number 3 if the correct choice is maybe, 0 means the question has no correct choice
  • controls: int -> control, default=empty – a mapping of choice numbers to control objects in the format 
      controls:
    1: { activate_milestone: ... }
    2: { deactivate_milestone: ... }
  ...
    When a user selects a choice which has a corresponding number, the milestones specified in the control object will be modified accordingly. Choices may be omitted. Valid numbers are in the range [1-max].

Question: multiple-choice

  • labels: list of strings – list of labels to display as options, must contain at least 2 labels
  • correct: list of ints – list of positions that correspond to the correct options, must contain at least 1 correct option
  • exact_match: bool, default=False – if True, specified milestone modifications occur only if the team selected all the correct options and no incorrect ones
  • controls: int -> control, default=empty – a mapping of choice numbers to control objects, same as in radio questions

Question: free-form

Note: this question type is not allowed for repeatable questionnaires.

  • related_milestones: list of strings, default=empty – list of related milestone names, these milestones will be shown to the instructor for evaluation, example: 
    related_milestones:
  - milestone_1
  - milestone_2
# or
related_milestones: [ milestone_1, milestone_2 ]
  • multiline: bool, default=False – changes whether the input text field should be multiline or single line
  • min: int, default=0 – optional field that specifies the minimum length of the answer to be accepted, 0 means that an empty answer will be accepted
  • max: int, default=-1 – optional field that specifies the maximum length of the answer to be accepted, -1 signifies no requirement
  • llm_assessment: llm assessment, default=empty -

Question: auto-free-form

  • correct_answer: string – the correct answer for this question
  • regex: bool, default=False – determines whether correct_answer is used as a regex during answer evaluation Note: if you need to match a literal backslash (\) in the answer, you must write 4× as many backslashes in the definition. Example: to match \ write \\\\ in correct_answer; to match \\ write \\\\\\\\.
  • correct: control, default=empty – milestone modifications triggered when the given answer is correct, milestone_condition and roles cannot be specified
  • incorrect: control, default=empty – milestone modifications triggered when the given answer is incorrect, milestone_condition and roles cannot be specified
  • multiline: bool, default=False – changes whether the input text field should be multiline or single line
  • min: int, default=1 – optional field that specifies the minimum length of the answer to be accepted
  • max: int, default=-1 – optional field that specifies the maximum length of the answer to be accepted, -1 signifies no requirement