Skip to content

Exercise log format

INJECT allows for export of logs from an exercise for analysis purposes. The log files use the JSONL format. Below is description of the structure and schema of logs.

Pseudonymization

All logs exported from the platform are pseudonymized.

All actions performed by a specific user can be tracked by looking for the same user ID. This ID is unique for the platform, which means that if the same user performed actions in two different exercises, the actions would have the same user ID.

To link pseudonymized data to exercise participants, the instructor must have access to the exercise and download a pseudonymization (coding) key, which is a CSV file with this format:

  • uuid - the user ID (pseudonym)
  • username - username for the user ID
  • first_name - first name for the user ID, may not be specified
  • last_name - last name for the user ID, may not be specified

Structure

- logs/
  - team-id/
      - uploaded-files/
        - files uploaded by this team during the exercise
    - action_logs.jsonl
    - milestones.jsonl
    - inject_states.jsonl
    - questionnaire_states.jsonl
    - email_threads.jsonl
    - instructor_comments.jsonl
    - sandbox_help_feedback.jsonl
  - team-id+1/
    ...
  - definition_files/
    - files used by the definition
  - exercise.jsonl
  - teams.jsonl
  - team_states.jsonl
  - instructors.jsonl
  - file_infos.jsonl
  - exercise_injects.jsonl
  - exercise_milestones.jsonl
  - exercise_tools.jsonl
  - exercise_questionnaires.jsonl
  - exercise_learning_objectives.jsonl
  - exercise_channels.jsonl
  - exercise_controls.jsonl
  - exercise_confirmations.jsonl
  - exercise_contents.jsonl
  - email_participants.jsonl

The team directories are named team-<id> where <id> is the id of the team. Both definition_files/ and uploaded-files/ contain the raw files, each named by the id of its file info, see file_infos.jsonl.

Description and format of individual files

Many of the fields described here do not contain a description. That is because these fields are direct copies of the fields described in the definition format, which can be found here.

Timestamps

All fields with the timestamp type are timestamps in ISO 8601 format.

Common objects

Control

Controls are usually referenced by control_id, but they are also embedded directly in some objects (e.g. in questionnaire repeatable or in question details). The object has the following format:

  • id: int – id of the control
  • milestone_condition: string
  • dnf_clauses: list of maps – parsed form of milestone_condition, a map from milestone name to bool per clause
  • activate_milestone: optional list of strings
  • deactivate_milestone: optional list of strings
  • roles: optional list of strings
  • definition_id: int – id of the definition this control belongs to

Exercise files

exercise.jsonl

Contains information about the exercise. The object has the following format:

  • id: int – id of the exercise
  • name: string
  • timestamp_created: timestamp
  • uuid: uuid
  • on_demand: bool – whether the exercise is an on-demand exercise
  • sandbox: bool – whether the exercise has the sandbox feature enabled
  • llm: bool – whether the exercise has LLM-backed features enabled
  • created_by_id: optional uuid – id of the user that created the exercise
  • config_id: int – id of the exercise config
  • definition_id: int – id of the definition this exercise was created from
  • states: list of state objects – these objects represent the state of an exercise for the teams, one for normal exercises, multiple for on-demand exercises
    • id: int – id of the exercise state
    • status: int, (0=NOT_STARTED, 1=PAUSED, 2=ERROR, 3=RUNNING, 4=STOPPED, 5=EXPIRED, 6=FINISHED) – the status of the exercise state
    • elapsed_s: int – number of in-exercise seconds that elapsed in this exercise state
    • start_time: optional timestamp – timestamp when this exercise state was started
    • finish_time: optional timestamp – timestamp when this exercise state was finished
    • allowed_start_from: optional timestamp – start of the interval in which this exercise state can be started
    • allowed_start_to: optional timestamp – end of the interval in which this exercise state can be started
    • exercise_id: int – id of the exercise this state belongs to
    • definition_id: int – id of the definition

teams.jsonl

Contains information about all the exercise teams. Each object has the following format:

  • id: int – id of the team
  • name: string – name of the team
  • role: string – role of the team, empty string if no role
  • show_overview: bool – whether the team has the exercise overview enabled
  • team_state_id: int – id of the team state of this team, the referenced team state can be found in team_states.jsonl
  • exercise_state_id: int – id of the exercise state for this team, see exercise.jsonl
  • exercise_id: int – id of the exercise the team belongs to
  • users: list of uuids – ids of the trainees assigned to this team

team_states.jsonl

Contains the team states of all the exercise teams. Each object has the following format:

  • id: int – id of the team state
  • achieved_score: int – score achieved by the team during the exercise, includes score from instructor comments
  • total_score: int – total achievable score during the exercise, does not include score from instructor comments
  • finish_time: optional timestamp – time when the team reached one of the final milestones
  • exercise_id: int – id of the exercise the team state belongs to

instructors.jsonl

Contains all instructors assigned to the exercise. Each object has the following format:

  • id: uuid – id of the user

exercise_injects.jsonl

Contains all injects for this exercise. Each object has the following format:

  • id: int – id of the inject
  • name: string
  • display_name: string
  • time: int
  • delay: int
  • organization: string
  • type: string, (info, email)
  • definition_id: int – id of the definition
  • target_id: int – id of the channel this inject is sent to, the referenced channel can be found in exercise_channels.jsonl
  • alternatives: list of alternative objects
    • id: int – id of the alternative
    • inject_id: int – id of the inject this alternative belongs to
    • details: alternative details depending on the inject type

info alternative details

  • name: string
  • type: string, (info)
  • content_id: int – id of the content of this alternative
  • control_id: int – id of the control of this alternative
  • overlay_id: optional int – id of the overlay, not present if the alternative has no overlay
  • confirmation_id: optional int – id of the confirmation, not present if the alternative has no confirmation

email alternative details

  • name: string
  • type: string, (email)
  • sender: string
  • subject: string
  • extra_copies: int
  • content_id: int – id of the content of this alternative
  • control_id: int – id of the control of this alternative
  • overlay_id: optional int – id of the overlay, not present if the alternative has no overlay

exercise_milestones.jsonl

Contains all milestones for this exercise, which are referenced by milestones in each team. Each object has the following format:

  • id: int – id of the milestone
  • name: string
  • display_name: string
  • description: string
  • tags: optional list of strings
  • file_names: optional list of strings
  • final: bool
  • initial_state: bool
  • score: int
  • definition_id: int – id of the definition
  • activity_id: optional int – id of the learning activity this milestone belongs to

exercise_tools.jsonl

Optional file, included only if the tools feature is enabled. Contains all tools for this exercise, which are referenced by action logs in each team. Each object has the following format:

  • id: int – id of the tool
  • name: string
  • display_name: string
  • category: string
  • tooltip_description: string
  • hint: string
  • button_caption: string
  • default_response_id: int – id of the content sent as the default response of the tool
  • roles: optional list of strings
  • requires_input: bool
  • definition_id: int – id of the definition
  • responses: list of response objects
    • id: int – id of the response
    • param: string
    • time: int
    • regex: bool
    • content_id: int – id of the content of this response
    • control_id: int – id of the control of this response
    • tool_id: int – id of the tool this response belongs to

exercise_questionnaires.jsonl

Optional file, included only if the questionnaires feature is enabled. Contains all questionnaires for this exercise. Each object has the following format:

  • id: int – id of the questionnaire
  • name: string
  • display_name: string
  • repeatable: optional repeatable object
    • max_attempts: int
    • on_fail: control
  • time: int
  • delay: int
  • show_feedback: bool
  • post_exercise_submission: bool
  • included_file_ids: optional list of uuids – ids of the file infos included with this questionnaire
  • content_id: int – id of the content of this questionnaire
  • control_id: int – id of the control of this questionnaire
  • overlay_id: optional int – id of the overlay of this questionnaire
  • definition_id: int – id of the definition
  • questions: list of question objects
    • id: int – id of the question
    • type: string, (radio, free-form, auto-free-form, multiple-choice)
    • note: string
    • content_id: int – id of the content of this question
    • questionnaire_id: int – id of the questionnaire this question belongs to
    • details: additional details depending on the question type

radio question details

  • type: string, (radio)
  • max: int
  • correct: int
  • labels: list of strings
  • controls: map from label index to control

multiple-choice question details

  • type: string, (multiple-choice)
  • labels: list of strings
  • correct: list of ints
  • controls: map from label index to control
  • exact_match: bool

free-form question details

  • type: string, (free-form)
  • related_milestones: optional list of ints
  • multiline: bool
  • min: int
  • max: int
  • llm_assessment_id: optional int – id of the LLM assessment used to evaluate the answer

auto-free-form question details

  • type: string, (auto-free-form)
  • correct_answer: string
  • regex: bool
  • multiline: bool
  • min: int
  • max: int
  • correct: control
  • incorrect: control

exercise_learning_objectives.jsonl

Contains all learning objectives for the exercise. Each object has the following format:

  • id: int – id of the learning objective
  • name: string
  • description: string
  • tags: optional list of strings
  • order: int
  • definition_id: int – id of the definition
  • total_score: int – total score achievable by completing all activities in this objective
  • activities: list of activity objects
    • id: int – id of the activity
    • name: string
    • description: string
    • tags: optional list of strings
    • objective_id: int – id of the objective this activity belongs to
    • type: string
    • total_score: int – total score achievable by reaching all milestones linked to this activity
    • milestone_ids: list of ints

exercise_channels.jsonl

Contains all channels that exist for the exercise. Each object has the following format:

  • id: int – id of the channel
  • name: string
  • display_name: string
  • description: string
  • type: string, (info, email, tool, form, command)
  • definition_id: int – id of the definition

exercise_controls.jsonl

Contains all controls for this exercise. The objects follow the format described in Control.

exercise_confirmations.jsonl

Contains all confirmations for this exercise. Each object has the following format:

  • id: int – id of the confirmation
  • text: string
  • control_id: int – id of the control executed when the confirmation is confirmed

exercise_contents.jsonl

Contains all contents of this exercise, i.e. contents created from the definition and contents created during the exercise (e.g. contents of emails and custom injects). Each object has the following format:

  • id: int – id of the content
  • raw: string – raw text, possibly markdown
  • rendered: string – raw text converted to html elements
  • attachments: optional list of uuids – ids of the file infos attached to this content
  • included_file_ids: optional list of uuids – ids of the file infos included in this content including embedding
  • definition_id: optional int – id of the definition, set for contents created from the definition
  • exercise_id: optional int – id of the exercise, set for contents created during the exercise

email_participants.jsonl

Optional file, included only if the email feature is enabled. Contains all email participants for this exercise, which are referenced by email threads and email action logs in each team. Each object has the following format:

  • id: int – id of the email participant
  • address: string
  • team_visible: bool
  • exercise_id: int – id of the exercise this participant belongs to
  • definition_address_id: optional int – id of the definition address this participant represents
  • team_id: optional int – team which this participant represents, null if it does not belong to any team
  • definition_address: optional definition address object – email address from the definition which this participant represents, null if it belongs to a team, definition address format:
    • id: int – id of the definition address
    • address: string
    • description: string
    • team_visible: bool
    • organization: string
    • signed: bool
    • definition_id: int – id of the definition
    • control_id: int – id of the control executed when this address receives an email
    • llm_assessment_id: optional int – id of the LLM assessment used to evaluate emails sent to this address

file_infos.jsonl

Contains all file infos for this exercise, both definition files and files uploaded during the exercise. Each object has the following format:

  • id: uuid – id of the file, this is also the name of the file on the file system
  • file_name: string – original name of the file
  • full_path: string – full path of the file within the definition or the drive
  • definition_id: optional int – id of the definition, set for files that belong to the definition
  • is_drive: bool – whether this is a drive file
  • uploaded_by_id: optional uuid – uuid of the user that uploaded this file, null if the file belongs to the definition
  • uploaded_at: timestamp – timestamp when this file was uploaded
  • exercise_id: optional int – id of the exercise, set for files uploaded during the exercise

Individual team files

action_logs.jsonl

Contains all the action logs for the team. Each object has the following format:

  • id: int – id of the action log
  • timestamp: timestamp – time when this action log was created
  • in_exercise_time: int – the in-exercise-time value when this action happened
  • requires_attention: bool – whether this action requires attention of an instructor
  • type: string, (INJECT, CUSTOM_INJECT, TOOL, EMAIL, FORM, FORM_SUBMISSION, FORM_REVIEW, CONFIRMATION, FILE_DOWNLOAD, MILESTONE_MODIFICATION, SANDBOX_LOG, SANDBOX_HELP) – type of the action log
  • details: additional details depending on the action log type, the type field is repeated in the details object
  • channel_id: int – id of the channel this action log was sent to
  • team_id: int – id of the team
  • user_id: optional uuid – optional uuid of the user that performed this action, null for automatic actions performed by the platform
  • previous_log_ids: optional list of ints – ids of the action logs that are connected to this action log

INJECT details

  • inject_id: int – id of the inject
  • alternative_id: int – id of the sent alternative
  • content_id: int – id of the sent content
  • overlay_id: optional int – id of the shown overlay
  • confirmation_id: optional int – id of the shown confirmation

CUSTOM_INJECT details

  • content_id: int – id of the sent content
  • overlay_id: optional int – id of the shown overlay

TOOL details

  • argument: string – argument provided to the tool
  • tool_id: int – id of the used tool
  • selected_response_id: optional int – id of the selected tool response, null when the default response was used
  • content_id: int – id of the sent content

EMAIL details

  • thread_id: int – id of the thread this email was sent to, the referenced thread can be found in email_threads.jsonl
  • content_id: int – id of the content of the email
  • sender_id: int – id of the email participant that sent the email, the referenced participant can be found in email_participants.jsonl
  • overlay_id: optional int – id of the shown overlay

FORM details

  • questionnaire_id: int – id of the sent questionnaire

FORM_SUBMISSION details

  • questionnaire_id: int – id of the answered questionnaire
  • attempt: int – attempt number of this submission
  • accepted: bool – flag that determines whether this submission was accepted by the platform, controlled by the repeatable field on the questionnaire
  • correctness: string, (correct, incorrect, partially_correct, unknown) – correctness of the whole submission, excluding questions which cannot be automatically evaluated
  • answers: list of answer objects
    • answer: list of strings – submitted answers for this question
    • correctness: string, (correct, incorrect, partially_correct, unknown) – correctness of the answer as determined by the backend
    • attempt: int – attempt number of this answer
    • question_id: int – id of the question

FORM_REVIEW details

This object currently contains no additional fields.

CONFIRMATION details

This object currently contains no additional fields.

FILE_DOWNLOAD details

  • file_info_id: uuid – uuid of the file info that was downloaded

MILESTONE_MODIFICATION details

  • activated_milestone_states: list of ints – ids of the milestone states that were activated
  • deactivated_milestone_states: list of ints – ids of the milestone states that were deactivated
  • cause: int, (0=Trainee action, 2=Instructor action, 4=Automatic action) – the cause for this milestone modification

SANDBOX_LOG details

  • @timestamp: timestamp – time when the command was executed
  • cmd: string – the executed command
  • cmd_source: string – the source of the log within the container (i.e., Filebeat)
  • working_directory: string – the directory within the container in which the command was executed
  • username: string – the user by whom the command was executed within the container
  • container: string – the name of the container in which the command was executed

SANDBOX_HELP details

One object represents one sandbox help request of the team and its outcome.

  • questionnaire_id: int – id of the questionnaire (task) the trainee requested help for
  • user_prompt: optional string – the trainee's free-form prompt attached to the request
  • answer: optional string – the generated answer, not present when the request was rate limited or the LLM request queue was full
  • context_from: optional timestamp – start of the window of the sandbox command history used for the request
  • context_to: optional timestamp – end of the window of the sandbox command history used for the request
  • rate_limited: bool – whether the request was refused by the per-team rate limit
  • retry_after_seconds: optional int – how many seconds the team has to wait before the next request, present only when rate_limited is true
  • llm_busy: bool – whether the request was refused because the LLM service was busy
  • failed: bool – whether the request failed
  • latency_ms: optional int – total time it took the platform to send the response
  • response_time_ms: optional int – time it took the LLM to generate the answer
  • model: optional string – name of the model that generated the answer

milestones.jsonl

Contains all milestone states for the team. Each object has the following format:

  • id: int – id of the milestone state
  • reached: bool – state of the milestone
  • score: int – score the team received for this milestone
  • timestamp_reached: optional timestamp – time of the last state change for this milestone
  • team_state_id: int – id of the team state this milestone state belongs to, the referenced team state can be found in team_states.jsonl
  • milestone_id: int – id of the milestone this state belongs to, the referenced milestone can be found in exercise_milestones.jsonl
  • activity_id: optional int – id of the learning activity this milestone state belongs to

inject_states.jsonl

Contains all the inject states for the team. Each object has the following format:

  • id: int – id of the inject state
  • status: int, (0=UNSENT, 2=DELAYED, 4=SENT) – status of the inject
  • release_time: int – in-exercise time when a delayed inject should be sent to the team, non-zero only when the inject was delayed
  • team_id: int – id of the team
  • inject_id: int – id of the inject
  • alternative_id: optional int – id of the sent alternative

questionnaire_states.jsonl

Optional file, included only if the questionnaires feature is enabled. Contains all the questionnaire states for the team. Each object has the following format:

  • id: int – id of the questionnaire state
  • status: int, (0=UNSENT, 1=DELAYED, 2=SENT, 4=ANSWERED, 6=REVIEWED) – status of the questionnaire
  • release_time: int – in-exercise time when a delayed questionnaire should be sent to the team, non-zero only when the questionnaire was delayed
  • attempts: int – number of attempts the team has used
  • team_id: int – id of the team
  • questionnaire_id: int – id of the questionnaire

Submissions are not stored in this file, each submission appears as a FORM_SUBMISSION action log in action_logs.jsonl.

email_threads.jsonl

Optional file, included only if the email feature is enabled. Contains all email threads for the team. Each object has the following format:

  • id: int – id of the email thread
  • subject: string
  • timestamp: timestamp – time when this email thread was created
  • participant_ids: list of ints – ids of the email participants that belong to the thread, the referenced participants can be found in email_participants.jsonl
  • exercise_id: int – id of the exercise

Individual emails are not stored in this file, each email appears as an EMAIL action log in action_logs.jsonl, which references the thread by thread_id.

instructor_comments.jsonl

Contains all instructor comments for the team. Each object has the following format:

  • id: int – id of the comment
  • comment: string – the content of the comment
  • score: int – score value assigned to the comment, this value is added to the team's achieved score, can be negative
  • created_at: timestamp – timestamp when this comment was created
  • created_by_id: optional uuid – uuid of the instructor that created this comment
  • edited_at: optional timestamp – timestamp when this comment was edited
  • edited_by_id: optional uuid – uuid of the instructor that edited this comment
  • team_id: int – id of the team
  • exercise_id: int – id of the exercise
  • action_log_id: int – id of the action log this comment belongs to

sandbox_help_feedback.jsonl

Optional file, included only if both LLM and Sandbox feature is enabled. Contains all feedback the team gave to sandbox help answers. Each object has the following format:

  • action_log_id: int – id of the SANDBOX_HELP action log this feedback belongs to
  • helpful: bool – whether the trainee marked the answer as helpful
  • created_at: timestamp – timestamp when this feedback was created
  • user_id: optional uuid – uuid of the trainee that gave the feedback

Comparing logs from multiple exercises

The logs are constructed in a way that should allow for simple comparison of logs from multiple exercises with the same definition. It has to be the exact same upload of the definition, otherwise the IDs of injects, tools and other definition data will not match.

Between different runs of the same definition, the exercise_injects.jsonl, exercise_milestones.jsonl, exercise_tools.jsonl, exercise_questionnaires.jsonl, exercise_channels.jsonl, exercise_controls.jsonl, exercise_confirmations.jsonl and exercise_learning_objectives.jsonl files will be the same. The email_participants.jsonl will not match because email participants need to be re-generated for every exercise. However, the definition address they are linked to will have the same ID in all runs.