198 terms from AI work, model evaluation, data annotation and the technology behind language models, explained in plain English. Each entry has a sentence you can use in an interview.
The same definitions appear when you hover or tap an underlined term anywhere on the site.
Interview practice
Say a short definition out loud, then show the answer and compare. The cards follow the category you picked above. Nothing is scored or saved.
A prompt written on purpose to confuse a model or push it into an error.
Adversarial prompts test weak points such as misleading premises, hidden constraints, tricky maths or harmful requests in disguise.
A good adversarial prompt looks natural and targets one specific weakness.
Related Red teaming Prompt writing
An AI system that plans and takes several steps on its own, using tools such as search, code or apps.
Agents take actions as well as talk. They browse, edit files or call services. Evaluating them means checking each step, the final result and whether the agent stayed within its permissions.
With agents I review the whole trajectory, every step and tool call, as well as the final message.
Related Tool use Trajectory Prompt injection
A chat product built on a language model that answers questions and helps with tasks.
Assistants combine a tuned model with a system prompt, safety layers and sometimes search, files or code tools. Much evaluation work rates assistant responses.
When I rate an assistant, I judge the response the user sees, whatever tools were used behind it.
Work that keeps AI systems from causing harm, from everyday misuse to serious long-term risks.
AI safety covers content policies, testing for dangerous capabilities, red teaming, monitoring and research into how models behave.
In evaluation work, safety means checking that a helpful answer cannot easily be used to hurt someone.
Related Alignment Red teaming Content policy
A common job title for people who write, rate or correct data that is used to train and test AI models.
AI trainers write prompts, compare model answers, rewrite weak responses and label data. The title covers many task types, from simple ratings to expert review.
As an AI trainer my job is to give the model clear, consistent examples of good and bad answers.
Related RLHF Data annotation
A writing habit that shows up again and again in generated text, such as stock phrases or slogan-like lines.
AI-isms include patterns like “it is not X, it is Y”, heavy dashes and empty intensifiers. They matter when they make text less clear. Who wrote the text is a separate question.
I flag an AI-ism when it adds no meaning, and I suggest a plainer sentence instead.
Related Verbosity
Making a model’s behaviour match human intentions and values, such as being helpful, honest and harmless.
Alignment work includes preference training, safety rules, evaluations and red teaming. Human raters are a central part of it.
I see rating work as practical alignment, because each judgment teaches the model what people actually want.
Related RLHF Harmlessness Red teaming
A defined way for programs to talk to each other, for example to send a prompt to a model and get the answer.
Model providers offer APIs that developers use in their products. Usage is often billed per token.
Products reach models through an API, so the same model can behave differently with different system prompts.
Related Tool use Rate limit
Computer systems that do tasks which usually need human intelligence, such as understanding language or recognising images.
Today most AI products use machine learning, where systems learn patterns from data instead of following hand-written rules.
In my work, AI usually means machine learning models, especially language models trained on large datasets.
Related Machine learning Large language model
The worker who does a task first, before a reviewer checks it.
Many platforms use two roles. Attempters complete tasks and reviewers check them. Good attempters often move up to reviewer roles.
As an attempter I write my reasoning so a reviewer can follow every decision quickly.
Related Reviewer
The part of a Transformer that lets each token draw information from other tokens in the text.
Attention computes how much each earlier token matters for the current one, for example which noun a pronoun refers to. It is one part of the calculation and no full explanation of an answer.
Attention helps the model connect words across a sentence, which is why it can resolve what “it” refers to.
Related Transformer Context window
A later check of a batch of finished tasks to measure quality and find repeated problems.
An audit usually samples many tasks at once, often from different workers. It looks for systematic errors, such as a misunderstood rule, more than for single mistakes.
When an audit finds a pattern in my work, I reread the related section of the guidelines and adjust all similar tasks.
Related Quality assurance Reviewer
Generating text one token at a time, each time feeding the text so far back into the model.
Every new token depends on all earlier ones, so an early mistake can steer the rest of the answer.
Since generation is autoregressive, I read whole answers, because an early error often carries through.
The method that computes how much each weight contributed to the error, working backwards through the network.
Backpropagation applies the chain rule layer by layer. Its gradients are then used by gradient descent.
Backpropagation works out which weights to change, and gradient descent changes them.
Related Gradient descent Loss
A group of examples processed together in one training step.
Larger batches give smoother updates and need more memory.
Training uses batches so that each update reflects many examples at once.
Related Epoch
A fixed set of tasks used to measure and compare models.
Benchmarks test skills such as maths, coding or reading. They make progress measurable but can become outdated or leak into training data, which inflates scores.
Benchmarks are useful for comparison, but I check real-world tasks too, since a high score can hide specific weaknesses.
Related Evaluation set Data contamination
A systematic tilt in data or outputs that treats some groups or answers unfairly or inaccurately.
Bias can come from unbalanced data, labeler assumptions or model training. Evaluators flag stereotypes, uneven treatment and one-sided framing.
I check whether a response would change if the person in the prompt had a different gender, age or background.
Related Fairness Training data
A rectangle drawn around an object in an image to show where it is.
Bounding boxes train object detection models. Guidelines define how tight the box must be, what to do with partly hidden objects and which objects to skip.
I keep boxes tight to the visible edges and follow the project rule for objects that are partly hidden.
Not the same as Segmentation, which outlines the exact shape instead of a rectangle.
Related Segmentation Object detection
A tokenizer method that repeatedly joins the most frequent neighbouring pieces of text.
BPE starts from characters or bytes and learns merges such as “t” plus “h” into “th”. Frequent words end up as single tokens, rare ones as several.
BPE is why common words cost one token and unusual names cost several.
Related Tokenizer
Reviewers compare their judgments on the same examples so that everyone applies the rules the same way.
In a calibration round, several people score the same tasks and then discuss the differences. It aligns how a team reads the guidelines and raises agreement between workers.
Calibration sessions help me see how others read the same rule, and I adjust my scores when the team agrees on a reading.
Related Inter-annotator agreement Guidelines
Step-by-step reasoning written out before a final answer.
Asking for or training chain of thought improves results on multi-step problems. Evaluation tasks may ask you to check each step.
When I check a chain of thought, I find the first wrong step and explain why it is wrong.
Related Reasoning model Prompt engineering
A program that talks with people in natural language, today usually powered by a language model.
Older chatbots followed scripts. Modern AI assistants generate answers, use tools and keep the conversation’s context.
A modern chatbot is a language model with a system prompt, tools and safety rules around it.
Related AI assistant Large language model
A saved version of a model’s weights at some point in training.
Teams compare checkpoints to pick the best one. Evaluation projects may compare responses from different checkpoints without telling raters which is which.
Side-by-side tasks often compare two checkpoints, so my honest preference helps choose the better version.
Related Side-by-side comparison
A reference that shows where a piece of information comes from.
Models sometimes invent citations or cite real sources that do not support the claim. Evaluators open the source and check it.
I open every citation in a response and check that it exists and actually supports the sentence it follows.
Related Hallucination Primary source
A single statement that can be checked as true or false.
Breaking text into claims makes checking manageable. Opinions and predictions are not checkable claims in the same way.
I split a response into separate claims and check the ones that matter most for the user.
Related Fact checking
The text a model generates in reply to a prompt.
Also called output or response. In evaluation tasks, completions are what you rate, compare or rewrite.
I judge a completion against what the prompt actually asked. What I would have written is beside the point.
The field of AI that works with images and video.
Computer vision covers classification, detection, segmentation, tracking and image generation. Much annotation work supports these models.
My image annotation experience is in computer vision tasks such as boxes and segmentation.
Related Object detection Segmentation
A specific requirement in a prompt, such as a word limit, a format or a word that must not appear.
Constraints make responses checkable. Prompt writers add them on purpose, and evaluators verify each one. Implicit constraints, such as the user’s language, count too.
I count words and check formats myself instead of trusting a response that claims it followed the limit.
Related Instruction following Prompt writing
Signed information attached to a file that records where it came from and how it was edited.
The C2PA standard lets cameras and editing tools sign images and videos. Missing credentials do not prove a file is fake.
Content credentials can confirm a file’s history, and their absence only means the history is unknown.
Related Provenance Watermarking
The written rules about what a model or platform may and may not produce.
Policies define harm categories, such as violence, self-harm or illegal activity, and the allowed exceptions, such as news or education. Safety tasks apply them case by case.
I apply the project’s content policy as written, including its exceptions for educational and news contexts.
Related Harmlessness Refusal
The maximum amount of text, in tokens, that a model can read at once, including the conversation so far.
Anything outside the context window is invisible to the model. Long conversations or documents may be cut, which can cause the model to forget earlier details.
If a model forgets an early instruction in a long conversation, the context window is one likely reason.
An error so serious that it makes a response fail regardless of its other qualities.
Typical critical errors are dangerous advice, a wrong final answer, ignoring the main instruction or leaking private data. Many rubrics cap the score when one appears.
If I find a critical error, I report it first and rate the response accordingly, even if the rest is strong.
The standard loss for classification and next-token prediction. It is low when the correct answer gets high probability.
Cross-entropy equals the negative log probability of the correct token, averaged over examples.
Cross-entropy rewards the model for putting high probability on the right next token.
Adding labels or notes to raw data, such as marking objects in a photo or tagging the topic of a sentence.
Annotated data teaches and tests models. The work ranges from simple tags to detailed expert labels, and it always follows project guidelines.
Annotation quality comes from consistency, so I apply the same rule to every similar example.
Not the same as Data collection, which gathers the raw data before anyone labels it.
Test or benchmark examples ending up in training data, which makes scores look better than real ability.
Contamination can happen when benchmarks are posted online and then scraped. Teams check for overlaps and keep private test sets.
I never paste evaluation tasks into public tools, partly to avoid contaminating future training data.
Related Benchmark Evaluation set
A collection of examples used to train, tune or test a model.
Datasets are usually split into training, validation and test parts. Their quality, balance and licensing shape the final model.
Every label I add becomes part of a dataset, so small inconsistencies can add up across thousands of examples.
Related Training data Evaluation set
Removing repeated or near-identical examples from a dataset.
Duplicates make models memorise and can leak test examples into training. Annotators are sometimes asked to flag near duplicates.
I flag near-duplicate items so they do not get extra weight in training.
Related Dataset Data contamination
Machine learning with neural networks that have many layers.
Deep learning powers modern language, image and speech models. More layers and data let models learn complex patterns, at a high computing cost.
Language models are deep learning systems, so their behaviour comes from training data more than from explicit rules.
Related Neural network Transformer
Realistic fake video, audio or images of real people, made with AI.
Deepfakes can spread lies or harm reputations. Verification uses source checks, reverse image search and content credentials.
With suspicious media I look for the original source first, before judging what it shows.
A model that creates images, video or audio by gradually removing noise from a random start.
Most image generators use diffusion. Evaluators check prompt adherence, anatomy, text in images and artefacts.
With generated images I check prompt adherence first, then visible artefacts such as hands and text.
Related Multimodal model Generative AI
A training method that learns directly from preference pairs, without a separate reward model.
DPO is simpler than classic RLHF and widely used. It still depends on the quality of human or AI preference data.
Whether a team uses RLHF or DPO, both learn from the preference judgments raters produce.
Related RLHF Preference data
Training a smaller model to imitate a larger one, so it becomes cheaper while keeping much of the quality.
The large teacher model produces outputs or probabilities, and the student learns from them.
Distilled models are cheaper to run, and their quality still needs human checks on hard cases.
Related Quantisation
Someone with real training or work experience in a field, such as law, medicine or coding.
Specialist projects ask experts to write hard questions, check model answers for subtle errors and explain them. They usually require proof of a degree or professional experience.
As a domain expert I check every claim against current sources in my field and explain errors in plain words.
Not the same as A generalist, who works on tasks that need no specialist knowledge.
Related Generalist Fact checking
Checking the public registration details of a website address, such as its creation date.
A very new domain can be a warning sign for a news-like site. Owner details are often hidden.
A domain lookup tells me how long a site has existed, which helps me judge an unknown source.
Related Lateral reading
Knowledge or help that can serve both good and harmful purposes.
Chemistry, security and biology questions are often dual-use. Policies decide how much detail a model may give and in which context.
For dual-use requests I check whether the response gives general understanding or step-by-step harmful detail.
Related Content policy Refusal
An unusual example that the main rules do not clearly cover.
Edge cases are where most disagreements and errors happen. Good guidelines list known edge cases. When you meet a new one, follow the closest rule and explain your choice in a comment.
When I meet an edge case, I apply the closest rule, explain my reasoning and flag it so the team can add it to the guidelines.
Related Guidelines Escalation
A list of numbers that represents a token, word or text, so that similar meanings get similar numbers.
Models turn tokens into embeddings before processing them. Separate embedding models turn whole texts into vectors for search and recommendation.
Embeddings let a system find texts with similar meaning even when they use different words.
Related Vector database Token
One full pass of training over the whole dataset.
Small fine-tuning sets are often used for a few epochs. Too many epochs can lead to overfitting.
An epoch means the model has seen every training example once.
Related Batch Overfitting
A project’s list of error types, such as factual error, missed instruction or formatting problem.
Labeling errors by type makes feedback comparable and shows the client which problems are most common. Each project defines its own taxonomy.
I use the project’s error categories exactly as defined, so my reports can be counted and compared.
Sending a task or question to a lead or project team because the guidelines do not answer it.
Projects usually have a channel for questions, such as a form, a chat or a flag button. Escalating unclear cases protects data quality and helps the team improve the guidelines.
I escalate a case when the guidelines truly do not cover it, and I include the task ID and the exact question.
Related Edge case
Examples kept apart from training and used only to measure how well a model performs.
If evaluation examples appear in training, scores look better than real performance. Teams keep evaluation sets private and refresh them over time.
I never reuse evaluation prompts in training tasks, because that would make the scores meaningless.
Related Benchmark Data contamination
Technical metadata stored in photos, such as camera model, date and sometimes location.
Many platforms strip EXIF data on upload, and it can be edited. It is a clue and never proof on its own.
EXIF data can support a date or place, and I confirm it with other evidence.
Related Metadata
A model saying how sure it is, for example “I am not certain” or “this depends on your country”.
Good responses match their confidence to the evidence. Too much hedging makes answers useless, and too little makes errors look certain.
I reward a response that flags real uncertainty and penalise one that hedges every sentence.
Related Honesty
Checking whether a claim is true by finding reliable evidence for or against it.
Fact checking separates a text into claims, finds the original source of each and compares them. Evaluators fact check model answers before rating accuracy.
I fact check by tracing each important claim back to its original source instead of another article that repeats it.
Related Primary source Claim Lateral reading
Whether the claims in a response are true according to reliable sources.
Factuality checks each claim separately. One false claim can make a long answer misleading. Evaluators check claims against primary or trusted sources and never against the model itself.
I break a response into separate claims and check the important ones against a primary source.
Related Hallucination Primary source
Treating people and groups equally and without unjustified bias in data and outputs.
Fairness checks look for stereotypes, uneven quality across languages or groups and one-sided framing.
I compare how a model answers the same question about different groups to spot unfair differences.
Related Bias
How closely a summary or answer reflects its source without adding, dropping or changing meaning.
A summary can be fluent and still unfaithful, for example by turning a possibility into a certainty. Faithfulness checks compare the output sentence by sentence with the source.
When I check a summary, I look for changed certainty, missing conditions and added details.
Related Grounding Factuality
Comments from reviewers or the project team about what you did well and what to change.
Good feedback names the guideline, the error and a better answer. On many projects it is the fastest way to improve, and acting on it raises your quality score.
I welcome feedback and I ask one clarifying question if a correction does not match my reading of the guidelines.
Related Reviewer Calibration
Including a few examples of the desired input and output in the prompt.
Examples show the model the format and style. They can also bias it toward copying details of the examples.
I choose few-shot examples that differ from each other, so the model learns the pattern and not one example.
Not the same as Zero-shot prompting, which gives no examples.
Further training of a pretrained model on a smaller, focused dataset to change its behaviour or skills.
Fine-tuning can teach a format, a domain or a style. It needs far less data than pretraining, so each example has more influence.
Because fine-tuning sets are small, a single bad example I write can teach the model the wrong habit.
Related Supervised fine-tuning LoRA
A large model trained on broad data that can be adapted to many tasks.
A base or foundation model comes out of pretraining. Fine-tuning and preference training turn it into an assistant or a specialised tool.
The assistant I evaluate started as a foundation model and was shaped by data like the ratings I produce.
Related Pretraining Fine-tuning
A model’s ability to perform well on new examples it did not see in training.
Good generalisation is the real goal of training. Varied, high-quality data and honest evaluation support it.
I write prompts on varied topics because the goal is a model that generalises beyond the training set.
Related Overfitting
A worker on tasks that need careful reading and judgment but no specialist field knowledge.
Many beginner projects are for generalists, such as rating responses, checking instructions or labeling everyday data. Strong reading, writing and attention to detail matter most.
As a generalist I focus on following instructions exactly and on writing clear reasons for my ratings.
Related Domain expert
AI that creates new content, such as text, images, audio, code or video.
Generative models learn patterns from examples and produce new outputs in a similar style. Human feedback shapes what counts as a good output.
Generative AI needs human judgment, because no automatic metric fully captures quality.
Related Large language model Diffusion model
Finding where a photo or video was taken by matching what it shows with maps and other images.
Signs, buildings, landscapes and shadows can be compared with satellite and street-level imagery.
To geolocate an image, I start with unique details such as signs or skyline shapes and match them on maps.
Related OSINT
Items with known correct answers, mixed into the work to check accuracy.
Gold tasks look like normal tasks. Your answers on them measure your quality. Treat every task as if it were a gold task.
I work the same way on every task, so hidden gold tasks simply confirm my normal quality.
Related Ground truth Quality assurance
A processor that does many calculations in parallel. Most AI training and inference runs on GPUs.
GPU time is a large part of the cost of AI. It is one reason providers limit usage and price by tokens.
Training large models takes thousands of GPUs, which is why every training example should be worth including.
The method that changes each weight a little in the direction that lowers the loss.
Gradients say how the loss changes with each weight. Optimisers such as Adam use them to update weights step by step.
Gradient descent is the small-step learning loop behind almost all model training.
Related Backpropagation Learning rate
Always choosing the single most likely next token.
Greedy decoding is predictable and can be repetitive. It equals sampling at temperature zero.
Greedy decoding gives the same answer every time for the same input.
Related Sampling
The correct answer or label that outputs are compared with.
Ground truth may come from experts, verified sources or agreement between several annotators. If the ground truth is wrong, every score based on it is wrong too.
When I suspect a gold label is wrong, I report it with evidence instead of quietly following it.
Basing an answer on provided documents or sources instead of the model’s memory.
In grounded tasks the response should use only the given context. Evaluators check that each claim appears in the source and that nothing was added from outside.
For grounded tasks I check every claim against the provided document and flag anything the document does not support.
Related RAG Faithfulness
Checks around a model that block or correct unsafe inputs and outputs.
Guardrails can be filters, classifiers, rules in the system prompt or human review. They add safety and can cause over-refusals if too strict.
Guardrails catch many problems, and evaluators find the cases they miss or block wrongly.
Related Content policy Over-refusal
The project’s written instructions. They define the labels, the edge cases and what to do when you are unsure.
Guidelines are the most important document on any project. They describe the goal, the scoring rules, examples and exceptions. When the guidelines and your own opinion disagree, the guidelines win.
If my judgment and the guidelines point in different directions, I follow the guidelines and flag the case in a comment.
Related Rubric Edge case Onboarding
An output that sounds plausible and is false or unsupported, such as an invented fact, quote or source.
Models generate likely text, and likely text can be wrong. Hallucinations are common with names, numbers, dates, citations and niche topics, so evaluators verify them.
I verify names, numbers and citations in every response, because that is where hallucinations hide most often.
Not the same as A typo or style problem, which is a different and usually less severe error.
Related Factuality Grounding Fact checking
Whether a response avoids causing harm, such as dangerous instructions, harassment or privacy leaks.
Harmlessness is weighed together with helpfulness. A good model helps with legitimate requests and declines the harmful part of a request, ideally with a safe alternative.
I check harmlessness by asking who could be hurt if the response were used as written.
Related Refusal Content policy
How well a response meets the user’s real need, beyond the literal words of the request.
A helpful answer is correct, complete enough, clear and suited to the user. It balances detail with brevity and anticipates an obvious follow-up question.
For helpfulness I ask whether the user could act on this answer right away without asking again.
Related Honesty Harmlessness Instruction following
A model’s quality of saying only what it has reason to believe and admitting uncertainty or limits.
An honest response avoids invented facts, overconfident claims and false capability claims, such as pretending to browse the web. It says when it does not know.
I mark a response down for honesty when it states a guess as a fact or claims an action it cannot perform.
Related Hallucination Expressed uncertainty
A setup where people check or correct an AI system’s output before it is used.
Human-in-the-loop work keeps people in charge of important decisions. AI trainers, reviewers and moderators are all part of such loops.
I see my role as the human in the loop who catches what the model gets wrong before it reaches a user.
A written example of the best possible answer to a prompt, used to train or score models.
Ideal responses must be correct, complete and in the requested format. They often become supervised training examples, so small errors can be copied by the model.
I write an ideal response as if a demanding expert user will read every line.
Creating images from a text description with an AI model.
Evaluators check whether the image follows the prompt, looks coherent and avoids policy problems.
For image generation I list the prompt’s elements and check each one in the picture.
Related Diffusion model
Someone who works per task or per hour without an employment contract. Pay, hours and access depend on the project.
Most AI training work is offered on a contract basis. You choose when to work, and the platform decides how much work is available. There is usually no fixed salary, paid leave or guaranteed number of hours, and you handle your own taxes.
I understand this is contract work, so I plan my week around the project’s availability and keep my own records of hours and tasks.
Not the same as An employee, who has a contract with set hours, pay and benefits.
Related Project Onboarding
Running a trained model to get an output. The model’s weights do not change during inference.
Every chat reply is inference. It costs computing power per token, which is why providers price usage by tokens.
During inference the model only uses what it learned and what is in the context. It does not learn from our conversation.
Not the same as Training, which changes the weights.
Whether a response does exactly what the prompt asked, including format, length and constraints.
Instruction following is usually the first thing to check. A beautiful answer that ignores a word limit or the requested format still fails the user.
I start every evaluation by listing the explicit requirements of the prompt and checking each one.
Related Constraint Helpfulness
Fine-tuning that teaches a model to follow instructions across many kinds of tasks.
Instruction-tuned models answer requests directly instead of just continuing text. Diverse, high-quality instructions are the key ingredient.
Instruction tuning is what turns a text predictor into an assistant that answers the actual request.
Labeling what a user wants from a message, such as “cancel order” or “ask for a refund”.
Intent labels train chatbots and routing systems. One message can contain several intents, and guidelines say which one to choose.
When a message has several intents, I follow the project rule, usually the main action the user asks for.
Related Sentiment analysis Labeling
How often different annotators give the same label to the same item.
High agreement shows clear guidelines and careful work. Low agreement points to unclear rules or hard examples. Statistics such as Cohen’s kappa measure it.
When agreement drops on a label, I check whether the guidelines leave room for two readings and raise it.
Related Calibration Gold standard
A trick that gets a model to ignore its safety rules, for example through role-play or hidden instructions.
Jailbreaks are studied in authorised red teaming so they can be fixed. Sharing working jailbreaks publicly can cause harm.
I only test jailbreak techniques inside an authorised project and report them through the official channel.
Not the same as Prompt injection, which hides instructions in content the model reads.
Related Red teaming Prompt injection
A common text format for data, made of keys, values, lists and nested objects.
Many tasks, tools and APIs use JSON. Small errors such as a missing quote or comma make it invalid.
I validate JSON before judging it, because one missing comma breaks the whole object.
Related Structured output API
The short written reason for your rating or preference.
A strong justification names the specific problem, quotes or points to it and says why it matters. Reviewers and training teams rely on it to trust and reuse your rating.
Each justification I write says what is wrong, where it is and why it matters for the user.
Related Side-by-side comparison Feedback
Marking specific points on an object, such as joints on a body or corners of a face.
Keypoints train pose and face models. Guidelines define the order of points and what to do when a point is hidden.
I follow the point order exactly, since swapped left and right points are a common error.
Related Bounding box
The date after which a model has no training data, so it does not know later events.
Without search tools, a model cannot know what happened after its cutoff. Good responses say so instead of guessing.
For recent events I check whether the model admits its knowledge cutoff or invents an answer.
Stored attention keys and values from earlier tokens, reused to make generation faster.
Without a cache, the model would recompute every earlier token for each new one. The cache uses memory, which limits long contexts.
The KV cache is an engineering trick for speed and does not change what the model knows.
Related Autoregressive generation Latency
Assigning a category or tag to a piece of data, such as “spam” or “not spam”.
Labeling is the simplest form of annotation. Label lists are defined in the guidelines, and unclear examples are handled by edge-case rules.
Before labeling, I read the full label list, because similar labels often differ by one small condition.
Related Data annotation Taxonomy
A model trained on huge amounts of text to predict the next token, used for chat, writing and many other tasks.
LLMs generate text one token at a time. After pretraining, they are tuned with human feedback to follow instructions and behave helpfully and safely.
An LLM predicts likely text, so I always check its claims instead of trusting its fluency.
Related Token Transformer Pretraining
The delay between sending a request and getting the answer.
Latency depends on model size, output length, server load and network. Products balance it against quality.
Latency matters for real products, so a shorter correct answer can be the better one.
Related Inference
Checking a source by leaving it and reading what other independent sources say about it.
Professional fact checkers judge an unknown site by searching for its owner, funding and reputation elsewhere.
With an unfamiliar website I read laterally: I search what others say about it before trusting it.
Related Fact checking
How big each training step is. Too high makes training unstable, too low makes it slow.
Learning rates are often warmed up at the start and lowered near the end of training.
The learning rate controls how fast a model changes in response to its errors.
Related Gradient descent
A rating scale with a fixed number of steps, such as 1 to 5 or 1 to 7, often from much worse to much better.
Evaluation tasks use Likert scales for overall quality or preference strength. Each step should have a written meaning in the rubric, so different reviewers use it the same way.
I use the extreme points of a Likert scale only for clear cases, and I check the rubric’s description for each level.
Related Rubric Side-by-side comparison
Using a language model to score or compare other models’ outputs.
Model judges are fast and cheap, and they have biases, such as preferring longer answers or their own style. Human reviewers check and calibrate them.
Automatic judges help with scale, and human review is still needed to catch their blind spots.
Related Benchmark Calibration
The raw scores a model gives every token in its vocabulary before they become probabilities.
Softmax turns logits into probabilities that add up to one. Temperature divides the logits before softmax.
Logits are the model’s raw preferences, and softmax turns them into probabilities.
Related Softmax Temperature
A cheap fine-tuning method that trains small added matrices instead of all the model’s weights.
LoRA makes it possible to adapt large models on modest hardware and to switch between several adapters.
LoRA is a popular way to fine-tune large models without retraining all of their weights.
Related Fine-tuning
A number that measures how wrong a model’s predictions are during training. Training tries to lower it.
For language models the usual loss is cross-entropy on the next token. A falling loss shows learning, and a very low loss on training data can mean memorisation.
Loss tells engineers how well the model fits its data, while human evaluation tells them whether the answers are actually good.
Related Cross-entropy Overfitting
Methods that let computers learn patterns from examples instead of following fixed rules.
A machine learning model adjusts its parameters on training data to make better predictions. Supervised learning uses labeled examples, unsupervised learning finds structure without labels.
Labeled data is the fuel of supervised machine learning, which is why annotation quality matters so much.
Related Deep learning Training data
An open standard for connecting AI assistants to tools and data sources in a common way.
MCP lets one integration work with many AI apps, for example to read files, query a database or use a calendar.
MCP standardises how assistants connect to tools, so a tool written once works in many apps.
Data about data, such as the date, author, device or location of a file.
Metadata helps sort and verify data. In fact checking, photo metadata can hint at when and where a picture was taken, though it can be edited or stripped.
I use metadata as a clue and confirm it with other evidence, since it can be changed.
Related EXIF data Provenance
False or misleading information. Disinformation is the same when spread on purpose.
Models can repeat misinformation from training data or invent it. Evaluators check claims, especially on health, elections and money.
For health or legal claims I check an official source before rating the response as accurate.
Related Hallucination Fact checking
A model design where only some sub-networks, the experts, are used for each token.
MoE models can have very many parameters while using only part of them per token, which lowers the cost of running them.
Mixture of experts is a way to make large models cheaper to run per token.
Related Parameter Transformer
A model that works with more than one type of data, such as text, images, audio or video.
Multimodal models can describe images, answer questions about charts or transcribe speech. Evaluating them requires checking the visual or audio content too.
When I evaluate a multimodal answer, I check the image myself before judging the description.
Related Computer vision Large language model
Finding and labeling names in text, such as people, organisations, places and dates.
NER annotation marks the exact span of each entity and its type. Guidelines define tricky cases, such as titles, nested names and abbreviations.
In NER tasks I check span boundaries carefully, for example whether a title belongs inside the person entity.
A contract that forbids you to share project details, tasks or client names outside the project.
Most AI projects ask workers to sign an NDA. It usually covers task content, guidelines, screenshots and the client’s identity. Breaking it can end your access and have legal consequences.
I respect NDAs, so I describe my past projects by task type and skills instead of client names or examples.
Related PII
A model made of many connected units that turn numbers into other numbers through learned weights.
Neural networks are loosely inspired by the brain. Each layer transforms its input, and training adjusts the weights so the final output becomes useful.
I think of a neural network as a very large function whose weights were tuned on examples.
Related Parameter Deep learning
A computer vision task in which a model finds objects in an image and draws boxes around them.
Object detection models learn from images annotated with bounding boxes and class labels.
Clean, consistent boxes matter because the detection model learns exactly the boundaries we draw.
Related Bounding box Computer vision
Turning text in images or scans into editable text.
OCR tasks ask workers to correct machine-read text or to mark text regions. Typical problems are similar letters, handwriting and layout.
When I correct OCR output, I compare it character by character with the image for numbers and names.
Related Computer vision
The project-specific training and tests you complete before you get real tasks.
After a platform accepts you, each project has its own onboarding. It usually includes reading the guidelines, watching short lessons and passing a few practice tasks or a quiz. Your onboarding results often decide which tasks you can see.
During onboarding I read the guidelines twice and keep notes on edge cases, because the practice tasks usually test exactly those.
Not the same as Screening, which decides whether a platform accepts you at all.
Related Screening Guidelines Qualification test
Software whose source code is public under a licence that lets others use, change and share it.
In AI the term is used loosely. Many “open” models publish weights without training data or code.
I distinguish open-source code from open-weight models, since the licences and what is shared differ.
Related Open-weight model
A model whose trained weights are published, so others can run and adapt it.
Open-weight models can be run locally and fine-tuned. Their licences vary, and publishing weights is different from publishing training data or code.
Open-weight means the model can be downloaded and run, while the training data may still be private.
Related Fine-tuning Parameter
Research that uses publicly available information, such as maps, archives and public records.
OSINT methods help verify places, dates and events. Ethical use avoids profiling private people.
I use open sources to verify claims about places and events, and I avoid collecting data on private people.
Related Geolocation Web archive
A refusal of a request that was actually safe to answer.
Over-refusals frustrate users and reduce helpfulness, for example refusing a medical question that asks for general information. Evaluators flag them as errors.
I flag over-refusal when a safe, common question gets a refusal or a warning instead of an answer.
Related Refusal Helpfulness
When a model learns its training examples too closely and does worse on new data.
An overfit model memorises instead of generalising. Held-out evaluation sets reveal it.
Diverse prompts help prevent overfitting, which is why projects ask for varied topics and styles.
Not the same as Underfitting, where the model has not learned the patterns well enough.
Related Generalisation Evaluation set
Judging two items against each other instead of scoring each one alone.
People find it easier to say which of two answers is better than to give an exact score. Many preference datasets and model rankings are built from pairwise comparisons.
Pairwise judgments help me stay consistent, because I compare concrete differences instead of guessing an absolute score.
A number inside a model that training adjusts. Large models have billions of them.
Parameters, also called weights, store what the model learned. Parameter count is one rough measure of model size and does not by itself show quality.
Parameter count tells me the model’s size, and I judge its quality from its outputs.
Related Neural network Training
Checking of a research paper by independent experts before publication.
Peer review raises reliability without guaranteeing correctness. Preprints are shared before peer review.
I note whether a study is peer reviewed or a preprint when I rely on it.
Related Preprint Primary source
Information that can identify a person, such as a name with an address, an ID number or a phone number.
Projects often ask workers to remove or flag PII and never to enter their own or others’ personal data into tasks.
I never add real personal data to prompts, and I flag any PII I find in task content.
A 3D set of points, often from a LiDAR sensor, that describes the shapes around a car or robot.
Point cloud annotation places 3D boxes around cars, people and obstacles. It is common for self-driving and robotics projects.
I check 3D boxes from several viewing angles, since a box that fits from above can be wrong from the side.
Related Bounding box
Outlining an object with many connected points to follow its shape.
Polygons are more precise than boxes and faster than pixel-level masks. They are common for buildings, products and irregular shapes.
With polygons I place points at every real corner and avoid cutting through the object.
Related Segmentation Bounding box
Pairs or rankings of responses with a human judgment about which is better.
Preference data trains reward models and methods such as DPO. Clear preferences with written reasons are more valuable than quick guesses.
Good preference data needs a clear winner and a reason, so I mark near ties honestly as near ties.
Related Side-by-side comparison RLHF Direct preference optimisation
A research paper shared publicly before peer review, for example on arXiv.
Preprints spread findings quickly, and some contain errors that review would catch. Many AI papers appear first as preprints.
I treat preprint results as early findings and look for later confirmation.
Related Peer review
The first and largest training stage, where a model learns to predict the next token on huge amounts of text.
Pretraining teaches language, facts and patterns, together with the errors and biases in the data. The result is a base model that is not yet a helpful assistant.
Pretraining gives a model broad knowledge, and later human feedback teaches it how to use that knowledge helpfully.
Related Foundation model Fine-tuning
The original evidence, such as a study, a law, an official statistic or a first-hand record.
Primary sources are the strongest evidence. News articles and summaries are secondary sources that can add errors.
Whenever I can, I verify a number in the original study or official dataset instead of an article about it.
Not the same as A secondary source, which reports on or interprets the original.
Related Secondary source Citation
People’s right to control information about themselves.
AI work touches privacy in training data, task content and model outputs. Laws such as the GDPR set rules for handling personal data.
I treat every real-looking name and address in a task as private, unless the guidelines say otherwise.
Related PII
One client job on a platform, with its own guidelines, task type, pay and time limits.
A platform can run many projects at once. Each project has its own instructions and reviewers. Projects open and close often, so the work available to you changes over time.
When I join a new project, I start with a few tasks at a careful pace and read the reviewer feedback before I speed up.
Related Guidelines Task Independent contractor
The text a person gives a model, such as a question or an instruction.
A prompt can include instructions, examples, documents and constraints. Clear prompts produce more useful and more checkable answers.
I read the prompt first and list its requirements before I look at any response.
Related System prompt Prompt engineering
Designing prompts carefully to get better, more reliable model outputs.
Techniques include clear instructions, examples, structure, role descriptions and asking for step-by-step reasoning.
In prompt engineering I change one thing at a time, so I can see what actually improved the output.
Related Few-shot prompting Chain of thought Prompt
Hidden instructions inside a web page, file or email that try to take control of an AI system reading it.
Agents that browse or read documents can be tricked by text such as “ignore your previous instructions”. Systems defend by treating such content as data.
When I test agents, I check whether they follow instructions found inside documents, which they should treat as data.
A task type where you write realistic requests for a model, often designed to test a specific skill.
Projects ask for prompts that are natural, clear and sometimes hard on purpose. Good prompts have one clear goal, the needed context and checkable requirements.
When I write prompts, I add one or two checkable constraints so the response can be judged objectively.
Related Prompt Adversarial prompt
Where a piece of content came from and how it changed along the way.
Provenance checks ask who made a file or claim first, when and in what context.
I establish provenance first, because the same image can be accurate in one context and misleading in another.
A short test that checks whether you can do a certain type of task before you get access to it.
Platforms use qualification tests to unlock projects or task types. They often mirror real tasks, such as rating two responses or labeling a few images, and are scored against the project guidelines.
I treat a qualification test like a real task. I read the instructions fully, check every requirement and only then submit.
Related Screening Gold standard
The checks that keep a project’s output consistent and correct, such as reviews, audits and gold tasks.
QA combines several methods. Reviewers check samples, hidden gold tasks test accuracy and audits look at patterns over time. The goal is data the client can trust.
I think of QA as part of my own work, so I check my tasks against the guidelines before anyone else does.
Related Reviewer Gold standard Audit
Storing a model’s weights with fewer bits so it runs faster and uses less memory.
Quantised models may lose a little quality, especially on hard tasks.
Quantisation trades a little accuracy for speed and lower cost.
Related Distillation Parameter
A setup where a system first finds relevant documents and then has the model answer using them.
RAG gives models current or private information without retraining. Quality depends on finding the right documents and on the model sticking to them.
With RAG answers I check whether the right documents were found and whether the answer stays faithful to them.
Related Grounding Vector database Retrieval
A cap on how many requests or tokens an account can use in a period of time.
Rate limits protect services from overload. Agents and batch jobs must handle them gracefully.
A rate limit error means too many requests in a short time. The prompt itself is fine.
Related API
A model trained to work through a problem step by step before giving its final answer.
Reasoning models spend more tokens thinking and often score higher on maths, coding and logic. Their reasoning can still contain errors, so evaluators check the steps.
With reasoning models I check the steps as well as the final answer, since a right answer can rest on a wrong step.
Related Chain of thought
Deliberately trying to make a model fail or behave badly, so problems are found before users find them.
Red teamers write tricky, adversarial or misleading prompts and report what goes wrong. Findings become training data and new safety rules.
In red teaming I document each attempt, the exact prompt and the failure, so engineers can reproduce it.
Related Jailbreak Adversarial prompt
A link from someone who already works with a platform. It may earn them a reward and never guarantees you a place.
Platforms sometimes reward workers who invite others. A referral link does not change the screening or the decision. Honest sites mark such links clearly.
I applied through the platform’s own page and passed the standard screening.
Related Screening
A response in which the model declines to help with a request.
Some refusals are correct, for clearly harmful requests. Others are unnecessary. A good refusal is brief, explains the limit without lecturing and offers safe help where possible.
I judge a refusal by whether the request really needed it and whether the model still helped with the safe part.
Related Over-refusal Harmlessness
A task where you improve or fully rewrite a model’s answer so it becomes an ideal example.
Rewritten responses become training examples, so they must be correct, complete and in the requested style. Check every fact and keep the user’s request at the centre.
When I rewrite a response, I fix the errors first, then improve structure and tone, and I check every fact I add.
Related Supervised fine-tuning Ideal response
Building and using AI in ways that are safe, fair, transparent and accountable.
Responsible AI covers data practices, testing, documentation, human oversight and clear communication with users.
For me, responsible AI means honest data work. I follow the guidelines, flag problems and protect privacy.
Related AI safety Human in the loop
Finding the documents or passages most relevant to a query.
Retrieval can use keywords, embeddings or both. Relevance tasks ask raters to judge how well results match a query.
In relevance tasks I judge whether a result answers the query’s intent. Shared words alone are not enough.
Related RAG Semantic search
A search that starts from a picture and finds pages that show the same or a similar image.
It helps find where an image first appeared, its original context and whether it was edited.
Before I trust a viral photo, I run a reverse image search to find its earliest version.
Related Provenance Deepfake
A person who checks other workers’ tasks, scores them and gives feedback.
Reviewers are usually experienced workers on the same project. They compare submitted tasks with the guidelines, correct errors and write feedback. Their scores often affect your access to tasks.
I read reviewer feedback carefully and keep a short list of my repeated mistakes, so I can fix them in the next tasks.
Related Quality assurance Audit Feedback
A model finding ways to score high on its reward without doing the task well.
Examples include padding answers because longer answers scored higher, or sounding confident to please raters. Careful rubrics and varied evaluation reduce it.
I avoid rewarding length or confidence by themselves, since that teaches models to game the score.
Related Reward model Verbosity
A model trained on human preferences to score how good a response is.
The reward model stands in for human judges during reinforcement learning. If preference data is inconsistent, the reward model learns the wrong lessons.
A reward model learns from raters like me, so careless ratings would teach it the wrong idea of quality.
Related RLHF Reward hacking
Like RLHF, with preferences produced by an AI model instead of people.
AI feedback scales cheaply. People still write the principles, check samples and fix the judge’s blind spots.
AI feedback helps with scale, and human reviewers keep it honest.
Related RLHF LLM-as-a-judge
A training method that uses human preferences between responses to teach a model which answers people find better.
People compare model responses, a reward model learns their preferences, and reinforcement learning tunes the main model to score well on that reward. Side-by-side ratings are a typical source of the data.
My side-by-side ratings are the kind of human feedback RLHF uses, so I keep my preferences consistent and well explained.
Related Reward model Preference data Direct preference optimisation
The project’s written scoring rules. They say what counts as an error and how serious it is.
A rubric turns quality into checkable criteria, such as accuracy, instruction following and style, each with levels. It keeps scores consistent between reviewers.
I score against the rubric line by line, and my comment names the criterion each problem breaks.
Related Severity Guidelines Likert scale
Choosing the next token at random according to its probability, instead of always taking the likeliest one.
Sampling gives varied outputs, so the same prompt can produce different answers. Greedy decoding always takes the top token.
Because of sampling, I may see different answers to the same prompt, so I judge each response on its own.
Related Temperature Greedy decoding
The checks a platform runs before it accepts you, such as a profile review, a test or a short interview.
Screening can include an identity check, a language or reasoning test, a writing sample, a domain test or a video interview. Each platform runs its own steps, and results often take days or weeks.
In screening tests I explain my reasoning briefly and stick to the instructions, since reviewers check how I follow rules as much as the answer.
Not the same as Onboarding, which comes after acceptance and belongs to one project.
Related Qualification test Onboarding
A short command such as site: or filetype: that narrows a Google search.
Operators help find official documents, exact phrases or pages on one domain. Some are documented, others unofficial or retired.
I use operators like site: and quotation marks to reach official sources quickly.
Related Fact checking
A source that reports on, summarises or interprets original evidence, such as a news article or review.
Secondary sources help find and understand primary ones. Check them against the original when accuracy matters.
I use secondary sources to find the original evidence, then check the claim there.
Related Primary source
Marking the exact pixels that belong to an object or a region in an image.
Semantic segmentation labels every pixel with a class, such as road or sky. Instance segmentation also separates individual objects, such as each car.
Segmentation needs patience at the edges, so I zoom in on boundaries and check overlaps between objects.
Related Bounding box Polygon annotation
Search that matches meaning instead of exact words, usually using embeddings.
A query about “cheap flights” can find a page about “low-cost airfares”. It powers RAG and modern site search.
Semantic search finds texts with the same meaning even when the wording differs.
Related Embedding Vector database
Labeling text as positive, negative or neutral, sometimes with finer emotions.
Sentiment is often ambiguous because of sarcasm, mixed opinions or context. Guidelines define how to treat such cases.
For sentiment I label the writer’s attitude toward the target named in the task. My own reaction stays out of it.
Related Labeling Intent classification
How much an error hurts the result. Each project defines its own levels, often from minor to critical.
A wrong medical dose and a missing comma are both errors, with very different severity. Projects define levels so that serious problems weigh more in the final score.
I rate severity by the effect on the user, so a factual error in the main answer counts more than a style issue.
Related Rubric Error taxonomy
A task where you compare two responses to the same prompt and decide which is better and by how much.
Side-by-side tasks produce preference data for training. You usually rate each response first, then choose a preference and explain it with concrete differences.
In side-by-side tasks I name the one or two differences that decide my preference, such as a factual error or a missed instruction.
Related Preference data Pairwise comparison
A function that turns a list of scores into probabilities that add up to one.
Higher scores get more probability. Softmax appears at the output of language models and inside attention.
Softmax is the step that turns scores into the probabilities a model samples from.
Related Logits
A continuous piece of text that you select and label, such as a name or a phrase.
Span tasks require exact start and end points. Including or missing a space or punctuation mark can count as an error.
I select spans precisely to the word boundaries the guidelines define.
Related Named entity recognition
Marking who speaks when in an audio recording.
Diarization separates speakers and their turns, including overlaps. It is common in meeting, interview and call-centre audio.
I mark speaker changes at the exact moment of the switch and label overlapping speech as the guidelines say.
Related Transcription Timestamp
Technology that turns spoken audio into written text.
ASR systems are trained and tested with human transcriptions. Accents, noise and names are common error sources.
When I correct ASR output, I listen for names, numbers and words the system swapped for similar sounds.
Related Transcription Text-to-speech
Model output in a strict format, such as JSON, that a program can read.
Structured outputs must be valid and complete. Evaluators check the format and the content separately.
With structured output I first check the format is valid, then whether each field is correct.
Related JSON Constraint
Fine-tuning on example prompts paired with ideal responses written or approved by people.
SFT shows the model what good answers look like. The examples come from tasks such as prompt writing, response writing and rewriting.
SFT data is only as good as its examples, so I treat every ideal response as something the model will copy.
Related Ideal response Instruction tuning
A model agreeing with the user or flattering them instead of giving an accurate answer.
Sycophantic models change a correct answer when the user pushes back, or praise flawed work. Evaluators reward responses that stay accurate and polite at the same time.
I mark sycophancy when a model drops a correct answer only because the user disagreed.
Related Honesty
Data generated by a model or program instead of collected from real people or events.
Synthetic data helps when real data is rare or private. It needs human checks, since generated examples can be repetitive or wrong.
When I review synthetic examples, I look for repetition and unrealistic details as well as errors.
Related Training data
Instructions from the product, usually hidden from the user, that set the model’s role, tone and rules.
System prompts tell the assistant how to behave, what to avoid and which tools it has. Evaluation tasks sometimes show them so you can check compliance.
When a task includes a system prompt, I check the response against both the system rules and the user request.
Related Prompt Instruction following
One unit of work in a project, such as rating a response, labeling an image or writing a prompt.
Tasks arrive in a queue. Each task has instructions, sometimes a time limit and a way to submit. Reviewers may check a sample of tasks and send feedback.
Before I submit a task, I compare it once more with the checklist in the guidelines.
Related Task queue Project
The list of tasks waiting for workers on a project. When it is empty, no work is available.
Task queues fill and empty depending on client demand. Some projects release tasks in batches. An empty queue is normal and does not mean you did anything wrong.
Queues change from day to day, so I keep an eye on more than one project when the platform allows it.
The structured list of categories a project uses, often arranged from general to specific.
A taxonomy defines which labels exist and how they relate. Changes to it are announced in guideline updates.
I keep the label taxonomy open while working, because similar subcategories are easy to mix up.
Related Labeling Error taxonomy
A setting that controls how random the next-token choice is. Low values make answers more predictable.
Temperature reshapes the probabilities before sampling. It changes how varied the output is and adds no knowledge. At very low temperature the model nearly always picks the likeliest token.
A lower temperature gives more consistent wording, and it does not make the facts more accurate.
Related Sampling Top-p sampling
Technology that turns written text into spoken audio.
TTS evaluation rates naturalness, pronunciation, rhythm and emotion. Raters often compare samples side by side.
In TTS tasks I listen for mispronounced names and unnatural pauses as well as overall naturalness.
Related Speech-to-text
The maximum or expected time for one task. Going far over or under it can raise a flag.
Projects set time limits to keep work efficient and fair. Very fast tasks may look careless and very slow ones may be unpaid beyond the limit. Read the project’s own rules.
I watch the expected time per task, and if a task needs more time for quality, I note why.
Related Task
A time code that marks where an event happens in audio or video.
Audio and video tasks often require start and end times for words, events or scenes. Precision rules differ between projects.
I scrub back and forth around boundaries so my timestamps match the real start of each event.
Related Transcription Video annotation
A small piece of text that a language model reads, often a word or part of a word.
Models read and write tokens instead of letters or whole words. Limits, prices and speeds are measured in tokens. A common English word is often one token, while rare words split into several.
Since models see tokens rather than letters, I double-check tasks that involve counting letters or spelling.
Related Tokenizer Context window
The tool that splits text into tokens and maps each token to a number.
Each model family has its own tokenizer and vocabulary. Methods such as byte-pair encoding learn frequent pieces from text.
Different tokenizers split the same word differently, which explains some odd model errors with names.
Related Token Byte-pair encoding
A model calling external tools, such as a calculator, search engine or API, to complete a task.
The model writes a structured request, the tool runs, and the result returns to the model. Evaluators check whether the right tool was used with the right inputs.
I check tool calls for correct arguments and whether the model used the tool result honestly.
A decoding setting that samples only from the k most likely next tokens.
Top-k removes very unlikely tokens before sampling. Small k values make outputs safer and more repetitive.
Top-k limits the candidates the model can pick, which reduces strange word choices.
Related Top-p sampling Sampling
A decoding setting that samples only from the smallest set of tokens whose probabilities add up to p.
With top-p of 0.9, the model ignores the unlikely tail of candidates. It is often used together with temperature.
Top-p and temperature both control variety in the output, and neither checks facts.
Related Top-k sampling Temperature
Rude, hateful, harassing or demeaning language.
Toxicity labeling and filtering protect users. Guidelines separate toxic content from discussion of toxic content, such as a quote in a news report.
I label toxicity by what the text does to its target, and I follow the project’s rules for quoted content.
Related Content policy Harmlessness
The process that adjusts a model’s weights on examples so that its predictions improve.
Training compares predictions with targets, measures the error and changes the weights a little, many millions of times. Large models are trained in stages, first pretraining, then fine-tuning and preference training.
My ratings feed the later training stages, where human judgment shapes how the model behaves.
Not the same as Inference, where the trained model is only used.
Related Pretraining Fine-tuning Loss
The examples a model learns from.
Training data can be raw text, labeled examples, conversations or preference pairs. Errors and biases in it are learned by the model.
I know my annotations become training data, so I avoid guessing and flag unclear items instead.
The full record of an agent’s steps, with its reasoning, tool calls and results from start to finish.
Trajectory evaluation finds where an agent went wrong, such as a bad plan, a wrong tool or a misread result.
When I grade a trajectory, I mark the first step that went wrong and explain how it affected the rest.
Related AI agent
Writing down exactly what a speaker says, in the format the project asks for.
Projects define whether to keep filler words, how to mark unclear speech, noise and overlaps and how to write numbers. Verbatim and clean transcription follow different rules.
In transcription I follow the style guide for fillers and unclear words instead of tidying speech on my own.
Related Speaker diarization Timestamp
The neural network design behind most language models, built around attention.
Introduced in 2017, the Transformer processes all tokens of a text in parallel and uses attention to relate them. Most current language models are decoder-only Transformers.
Most chat models today are Transformers, which predict each next token from all the tokens before it.
Related Attention Large language model
A database that stores embeddings and quickly finds the ones most similar to a query.
Vector databases are the memory behind many RAG systems.
A vector database lets a system look up passages by meaning in milliseconds.
Using more words than the task needs, such as repeated points, long introductions or filler.
Long answers can look thorough while hiding the key point. Many rubrics penalise unnecessary length, unless the user asked for detail.
I check whether each paragraph adds something the user needs, and I mark padding as verbosity.
Related AI-ism Helpfulness
Labeling objects, actions or events across the frames of a video.
Video tasks include tracking objects frame by frame, marking actions and describing scenes. Consistency across frames matters as much as accuracy in one frame.
In video tasks I check that each object keeps the same ID through the whole clip.
Related Timestamp Object detection
Hidden signals in generated content that let tools detect it was made by AI.
Watermarks can help label AI content, and they can be weakened by editing. They are one signal among several.
A missing watermark proves nothing, so I verify content through its source as well.
Related Content credentials Deepfake
A service that saves copies of web pages over time, so you can see what a page said earlier.
Archives help when a page was changed or deleted. Not every page or date is saved.
If a source changed, I check an archived copy to see what it said on the date in question.
Related Provenance
Using software, sometimes with AI, to run repeated business steps without manual work.
Examples are sorting emails, filling forms or drafting replies for review. Good automations keep a human check on important decisions.
I would automate the repetitive part of a workflow and keep a human review where errors are costly.
Related Human in the loop AI agent
Asking a model to do a task with instructions only, without examples.
Strong models handle many tasks zero-shot. Examples help when the format or judgment is unusual.
I try a clear zero-shot prompt first and add examples only if the format keeps going wrong.
Related Few-shot prompting
No term matches. Try a shorter word or another category.