The data source transformer is a good sustainable alternative if you do not want to manually change specific documents or upload static files with information to the bot. By adding transformer rules to a source those rules will be applied for all future updates of the sources, no matter if it's a webscrape or API-source.
Use cases
Replace and/or remove text
Modifying text in sources can help both the embedder service and the GPT model more easily retrieve sources and provide specific answers to questions.
Add synonyms
Add example questions
Remove conflicting information
Below is an example of how to add the rule of replacing the word 'recycle' with 'recycle, throw away or discard' as a way of adding synonyms to sources giving instructions on how to recycle.
If you want to add keywords in a new field based on a document ID you can use the following logic. <field_to_look_at> in this case would be the ID-field, and <field_to_write_to> is the new field with the keywords. X should be replaced with the document's ID and Y with the keywords you want that document to have.
```typescript
export type Value<T = any> = {
type: 'value';
value: T;
};
// Get from item
export type Get = {
type: 'get';
source: ExprString | string;
path: boolean;
};
// Get from scope
export type GetS = {
type: 'gets';
};
// Put [target]: value to output
export type Put = {
type: 'put';
target: Expr | string;
value: Expr;
};
// Slice an array.
export type Slice = {
type: 'slice';
from?: number; // Defaults to start of array
to?: number; // Defaults to end of array
value: ExprArray | ExprString;
};
// Select a single value by index | first | last on array or by key on object
export type SelectArray = {
type: 'select_array';
select: 'first' | 'last' | number;
value: ExprArray;
};
export type SelectObject = {
type: 'select_object';
select: string;
value: ExprObject;
};
export type Extract = {
type: 'extract';
extract: string;
value: ExprString;
};
// Split the argument string on `split`
export type Split = {
type: 'split';
split: string | string[];
trim?: boolean; // Defaults to true
value: ExprString;
};
// Apply `map` on each element in `value`. `map` can access the current element with GetS
export type Map = {
type: 'map';
value: ExprArray;
map: Expr;
};
// Flat `value`
export type Flat = {
type: 'flat';
value: ExprArray;
};
export type If = {
type: 'if';
left: Expr;
op?: 'eq' | 'and' | 'or'; // Defaults to 'eq'
right?: Expr; // Defaults to true
then: Expr;
otherwise?: Expr; // Defaults to []
};
// Simplified if (value contains expr => expr)
export type TakeItOrLeaveIt = {
type: 'take_it_or_leave_it';
value: ExprString;
expr: string | string[];
};
export type IsType = {
type: 'is_type';
of: 'string' | 'array';
value: Expr;
};
export type ToNumber = {
type: 'to_number';
value: ExprString;
};
export type StringComp = {
type: 'string_comp';
op: 'contains' | 'starts_with' | 'ends_with' | 'matches';
expr: string | string[];
value: ExprString;
};
export type StringManipulation = {
type: 'string_manipulation';
manipulation: 'to_lower' | 'to_upper' | 'trim';
value: ExprString;
};
export type Replace = {
type: 'replace';
value: ExprString;
replace: { from: string | ExprString; to: string | ExprString }[];
};
// 'Cartesian join' of `values`.
// Returns array of one or more of `values` results in an string array,
// otherwise, a single string
export type Concat = {
type: 'concat';
values: Expr[];
};
export type Sanitize = {
type: 'sanitize';
mode: 'html';
value: ExprString | ExprArray;
};
```
Templates to copy
Webscrape - replace text in content column
The code below will work for all web scrapes and is set up to replace text in the content column. All you need to to do is add text that should be removed and replaced in the replace section of the code.
