Data source transformer

The data source transformer allows you to add rules for transforming data from a source, as an alternative to modifying the original data source.

The data source transformer is a good sustainable alternative if you do not want to manually change specific documents or upload static files with information to the bot. By adding transformer rules to a source those rules will be applied for all future updates of the sources, no matter if it's a webscrape or API-source.

Use cases

Replace and/or remove text

Modifying text in sources can help both the embedder service and the GPT model more easily retrieve sources and provide specific answers to questions.

• Add synonyms

• Add example questions

• Remove conflicting information

Below is an example of how to add the rule of replacing the word 'recycle' with 'recycle, throw away or discard' as a way of adding synonyms to sources giving instructions on how to recycle.

{
  "type": "put",
  "value": {
    "type": "replace",
    "value": {
      "type": "get",
      "source": "content"
    },
    "replace": [
      {
        "from": "recycle",
        "to": "recycle, throw away or discard"
      }
    ]
  },
  "target": {
    "type": "value",
    "value": "content"
  }
}

Below are all possible transformations

```typescript
export type Value<T = any> = {
  type: 'value';
  value: T;
};

// Get from item
export type Get = {
  type: 'get';
  source: ExprString | string;
  path: boolean;
};

// Get from scope
export type GetS = {
  type: 'gets';
};

// Put [target]: value to output
export type Put = {
  type: 'put';
  target: Expr | string;
  value: Expr;
};

// Slice an array.
export type Slice = {
  type: 'slice';
  from?: number; // Defaults to start of array
  to?: number; // Defaults to end of array
  value: ExprArray | ExprString;
};

// Select a single value by index | first | last on array or by key on object
export type SelectArray = {
  type: 'select_array';
  select: 'first' | 'last' | number;
  value: ExprArray;
};

export type SelectObject = {
  type: 'select_object';
  select: string;
  value: ExprObject;
};

export type Extract = {
  type: 'extract';
  extract: string;
  value: ExprString;
};

// Split the argument string on `split`
export type Split = {
  type: 'split';
  split: string | string[];
  trim?: boolean; // Defaults to true
  value: ExprString;
};

// Apply `map` on each element in `value`. `map` can access the current element with GetS
export type Map = {
  type: 'map';
  value: ExprArray;
  map: Expr;
};

// Flat `value`
export type Flat = {
  type: 'flat';
  value: ExprArray;
};

export type If = {
  type: 'if';
  left: Expr;
  op?: 'eq' | 'and' | 'or'; // Defaults to 'eq'
  right?: Expr; // Defaults to true
  then: Expr;
  otherwise?: Expr; // Defaults to []
};

// Simplified if (value contains expr => expr)
export type TakeItOrLeaveIt = {
  type: 'take_it_or_leave_it';
  value: ExprString;
  expr: string | string[];
};

export type IsType = {
  type: 'is_type';
  of: 'string' | 'array';
  value: Expr;
};

export type ToNumber = {
  type: 'to_number';
  value: ExprString;
};

export type StringComp = {
  type: 'string_comp';
  op: 'contains' | 'starts_with' | 'ends_with' | 'matches';
  expr: string | string[];
  value: ExprString;
};

export type StringManipulation = {
  type: 'string_manipulation';
  manipulation: 'to_lower' | 'to_upper' | 'trim';
  value: ExprString;
};

export type Replace = {
  type: 'replace';
  value: ExprString;
  replace: { from: string | ExprString; to: string | ExprString }[];
};

// 'Cartesian join' of `values`.
// Returns array of one or more of `values` results in an string array,
// otherwise, a single string
export type Concat = {
  type: 'concat';
  values: Expr[];
};

export type Sanitize = {
  type: 'sanitize';
  mode: 'html';
  value: ExprString | ExprArray;
};
```