Data source transformer

The data source transformer allows you to add rules for transforming data from a source, as an alternative to modifying the original data source.

The data source transformer is a good sustainable alternative if you do not want to manually change specific documents or upload static files with information to the bot. By adding transformer rules to a source those rules will be applied for all future updates of the sources, no matter if it's a webscrape or API-source.

Note: The transformer is an advanced feature intended for users with coding experience. If you're not familiar with coding, we recommend starting with a pre-built template below.

Basics - Start here

Important: Your transformer code must include all the fields you want to import into Ebbot. include all the fields that you want imported into ebbot.

The image above shows an example CSV with three columns: id, question, and answer. To apply transformer code to this CSV, all of these columns must be included like in the code snippet below.

[
  {
    "type": "put",
    "value": {
      "type": "get",
      "source": "id"
    },
    "target": {
      "type": "value",
      "value": "id"
    }
  },
  {
    "type": "put",
    "value": {
      "type": "get",
      "source": "question"
    },
    "target": {
      "type": "value",
      "value": "question"
    }
  },
  {
    "type": "put",
    "value": {
      "type": "get",
      "source": "answer"
    },
    "target": {
      "type": "value",
      "value": "answer"
    }
  }
]

Use cases

Replace and/or remove text

Modifying text in sources can help both the embedder service and the GPT model more easily retrieve sources and provide specific answers to questions.

• Add synonyms

• Add example questions

• Remove conflicting information

Below is an example of how to add the rule of replacing the word 'recycle' with 'recycle, throw away or discard' as a way of adding synonyms to sources giving instructions on how to recycle.

{
  "type": "put",
  "value": {
    "type": "replace",
    "value": {
      "type": "get",
      "source": "content"
    },
    "replace": [
      {
        "from": "recycle",
        "to": "recycle, throw away or discard"
      }
    ]
  },
  "target": {
    "type": "value",
    "value": "content"
  }
}

Add keywords in new field based on document ID

If you want to add keywords in a new field based on a document ID you can use the following logic. <field_to_look_at> in this case would be the ID-field, and <field_to_write_to> is the new field with the keywords. X should be replaced with the document's ID and Y with the keywords you want that document to have.

{
  "type": "put",
  "value": {
    "type": "if",
    "left": {
      "type": "get",
      "source": "<field_to_look_at>"
    },
    "op": "eq",
    "right": {
      "type": "value",
      "value": "X"
    },
    "then": {
      "type": "value",
      "value": "Y"
    },
    "otherwise": {
      "type": "value",
      "value": ""
    }
  },
  "target": {
    "type": "value",
    "value": "<field_to_write_to>"
  }
}

Below are all possible transformations

```typescript
export type Value<T = any> = {
  type: 'value';
  value: T;
};

// Get from item
export type Get = {
  type: 'get';
  source: ExprString | string;
  path: boolean;
};

// Get from scope
export type GetS = {
  type: 'gets';
};

// Put [target]: value to output
export type Put = {
  type: 'put';
  target: Expr | string;
  value: Expr;
};

// Slice an array.
export type Slice = {
  type: 'slice';
  from?: number; // Defaults to start of array
  to?: number; // Defaults to end of array
  value: ExprArray | ExprString;
};

// Select a single value by index | first | last on array or by key on object
export type SelectArray = {
  type: 'select_array';
  select: 'first' | 'last' | number;
  value: ExprArray;
};

export type SelectObject = {
  type: 'select_object';
  select: string;
  value: ExprObject;
};

export type Extract = {
  type: 'extract';
  extract: string;
  value: ExprString;
};

// Split the argument string on `split`
export type Split = {
  type: 'split';
  split: string | string[];
  trim?: boolean; // Defaults to true
  value: ExprString;
};

// Apply `map` on each element in `value`. `map` can access the current element with GetS
export type Map = {
  type: 'map';
  value: ExprArray;
  map: Expr;
};

// Flat `value`
export type Flat = {
  type: 'flat';
  value: ExprArray;
};

export type If = {
  type: 'if';
  left: Expr;
  op?: 'eq' | 'and' | 'or'; // Defaults to 'eq'
  right?: Expr; // Defaults to true
  then: Expr;
  otherwise?: Expr; // Defaults to []
};

// Simplified if (value contains expr => expr)
export type TakeItOrLeaveIt = {
  type: 'take_it_or_leave_it';
  value: ExprString;
  expr: string | string[];
};

export type IsType = {
  type: 'is_type';
  of: 'string' | 'array';
  value: Expr;
};

export type ToNumber = {
  type: 'to_number';
  value: ExprString;
};

export type StringComp = {
  type: 'string_comp';
  op: 'contains' | 'starts_with' | 'ends_with' | 'matches';
  expr: string | string[];
  value: ExprString;
};

export type StringManipulation = {
  type: 'string_manipulation';
  manipulation: 'to_lower' | 'to_upper' | 'trim';
  value: ExprString;
};

export type Replace = {
  type: 'replace';
  value: ExprString;
  replace: { from: string | ExprString; to: string | ExprString }[];
};

// 'Cartesian join' of `values`.
// Returns array of one or more of `values` results in an string array,
// otherwise, a single string
export type Concat = {
  type: 'concat';
  values: Expr[];
};

export type Sanitize = {
  type: 'sanitize';
  mode: 'html';
  value: ExprString | ExprArray;
};
```

Templates to copy

Webscrape - replace text in content column

The code below will work for all web scrapes and is set up to replace text in the content column. All you need to to do is add text that should be removed and replaced in the replace section of the code.

[
  {
    "type": "put",
    "value": {
      "type": "get",
      "source": "id"
    },
    "target": {
      "type": "value",
      "value": "id"
    }
  },
  {
    "type": "put",
    "value": {
      "type": "get",
      "source": "url"
    },
    "target": {
      "type": "value",
      "value": "url"
    }
  },
  {
    "type": "put",
    "value": {
      "type": "get",
      "source": "title"
    },
    "target": {
      "type": "value",
      "value": "title"
    }
  },
  {
    "type": "put",
    "value": {
      "type": "replace",
      "value": {
        "type": "get",
        "source": "content"
      },
      "replace": [
        {
          "to": "<Add text here that will replace text below>",
          "from": "<Add text to remove here>"
        }
      ]
    },
    "target": {
      "type": "value",
      "value": "content"
    }
  },
  {
    "type": "put",
    "value": {
      "type": "get",
      "source": "excerpt"
    },
    "target": {
      "type": "value",
      "value": "excerpt"
    }
  }
]