# Advanced Transformer Settings

## What is the Advanced Transformer Settings?

The advanced transformer settings in EbbotGPT Knowledge is a tool designed for complex data needs. Because it requires precise configuration to maintain, we recommend exploring our standard Filters first to see if they meet your requirements.

<figure><img src="/files/JtQCbYQGAYQq9Fbup8Xo" alt=""><figcaption></figcaption></figure>

The data source transformer is a good sustainable alternative if you do not want to manually change specific documents or upload static files with information to the bot. By adding transformer rules to a source those rules will be applied for all future updates of the sources, no matter if it's a webscrape or API source.

Note: the transformer is an advanced feature intended for users with coding experience. If you're not familiar with coding, we recommend starting with a pre-built template.

## Using the Advanced Transformer&#x20;

Your transformer code must include all the fields you want to import into Ebbot – this is an important first step.

<figure><img src="/files/9rW3yoPRRKEb7NKB7XvX" alt=""><figcaption></figcaption></figure>

The image above shows an example CSV with three columns: `id`, `question`, and `answer`. To apply transformer code to this CSV, all of these columns must be included like in the code snippet below.

```
[
  {
    "type": "put",
    "value": {
      "type": "get",
      "source": "id"
    },
    "target": {
      "type": "value",
      "value": "id"
    }
  },
  {
    "type": "put",
    "value": {
      "type": "get",
      "source": "question"
    },
    "target": {
      "type": "value",
      "value": "question"
    }
  },
  {
    "type": "put",
    "value": {
      "type": "get",
      "source": "answer"
    },
    "target": {
      "type": "value",
      "value": "answer"
    }
  }
]
```

### Use cases on how to use the advanced transformer settings

#### Replace and/or remove text

Modifying text in sources can help both the embedder service and the GPT model more easily retrieve sources and provide specific answers to questions.

* Add synonyms
* Add example questions
* Remove conflicting information

Below is an example of how to add the rule of replacing the word 'recycle' with 'recycle, throw away or discard' as a way of adding synonyms to sources giving instructions on how to recycle.

```
{
  "type": "put",
  "value": {
    "type": "replace",
    "value": {
      "type": "get",
      "source": "content"
    },
    "replace": [
      {
        "from": "recycle",
        "to": "recycle, throw away or discard"
      }
    ]
  },
  "target": {
    "type": "value",
    "value": "content"
  }
}
```

#### Add keywords in new field based on document ID

If you want to add keywords in a new field based on a document ID you can use the following logic. \<field\_to\_look\_at> in this case would be the ID-field, and \<field\_to\_write\_to> is the new field with the keywords. X should be replaced with the document's ID and Y with the keywords you want that document to have.

```
{
  "type": "put",
  "value": {
    "type": "if",
    "left": {
      "type": "get",
      "source": "<field_to_look_at>"
    },
    "op": "eq",
    "right": {
      "type": "value",
      "value": "X"
    },
    "then": {
      "type": "value",
      "value": "Y"
    },
    "otherwise": {
      "type": "value",
      "value": ""
    }
  },
  "target": {
    "type": "value",
    "value": "<field_to_write_to>"
  }
}
```

### Below are all possible transformations that can be done

````
```typescript
export type Value<T = any> = {
  type: 'value';
  value: T;
};

// Get from item
export type Get = {
  type: 'get';
  source: ExprString | string;
  path: boolean;
};

// Get from scope
export type GetS = {
  type: 'gets';
};

// Put [target]: value to output
export type Put = {
  type: 'put';
  target: Expr | string;
  value: Expr;
};

// Slice an array.
export type Slice = {
  type: 'slice';
  from?: number; // Defaults to start of array
  to?: number; // Defaults to end of array
  value: ExprArray | ExprString;
};

// Select a single value by index | first | last on array or by key on object
export type SelectArray = {
  type: 'select_array';
  select: 'first' | 'last' | number;
  value: ExprArray;
};

export type SelectObject = {
  type: 'select_object';
  select: string;
  value: ExprObject;
};

export type Extract = {
  type: 'extract';
  extract: string;
  value: ExprString;
};

// Split the argument string on `split`
export type Split = {
  type: 'split';
  split: string | string[];
  trim?: boolean; // Defaults to true
  value: ExprString;
};

// Apply `map` on each element in `value`. `map` can access the current element with GetS
export type Map = {
  type: 'map';
  value: ExprArray;
  map: Expr;
};

// Flat `value`
export type Flat = {
  type: 'flat';
  value: ExprArray;
};

export type If = {
  type: 'if';
  left: Expr;
  op?: 'eq' | 'and' | 'or'; // Defaults to 'eq'
  right?: Expr; // Defaults to true
  then: Expr;
  otherwise?: Expr; // Defaults to []
};

// Simplified if (value contains expr => expr)
export type TakeItOrLeaveIt = {
  type: 'take_it_or_leave_it';
  value: ExprString;
  expr: string | string[];
};

export type IsType = {
  type: 'is_type';
  of: 'string' | 'array';
  value: Expr;
};

export type ToNumber = {
  type: 'to_number';
  value: ExprString;
};

export type StringComp = {
  type: 'string_comp';
  op: 'contains' | 'starts_with' | 'ends_with' | 'matches';
  expr: string | string[];
  value: ExprString;
};

export type StringManipulation = {
  type: 'string_manipulation';
  manipulation: 'to_lower' | 'to_upper' | 'trim';
  value: ExprString;
};

export type Replace = {
  type: 'replace';
  value: ExprString;
  replace: { from: string | ExprString; to: string | ExprString }[];
};

// 'Cartesian join' of `values`.
// Returns array of one or more of `values` results in an string array,
// otherwise, a single string
export type Concat = {
  type: 'concat';
  values: Expr[];
};

export type Sanitize = {
  type: 'sanitize';
  mode: 'html';
  value: ExprString | ExprArray;
};
```
````

## Templates to copy when using the advanced transformer settings

**Template:** Scrape site - replace text in content column

The code below will work for all web scrapes and is set up to replace text in the content column. All you need to to do is add text that should be removed and replaced in the replace section of the code.

```
[
  {
    "type": "put",
    "value": {
      "type": "get",
      "source": "id"
    },
    "target": {
      "type": "value",
      "value": "id"
    }
  },
  {
    "type": "put",
    "value": {
      "type": "get",
      "source": "url"
    },
    "target": {
      "type": "value",
      "value": "url"
    }
  },
  {
    "type": "put",
    "value": {
      "type": "get",
      "source": "title"
    },
    "target": {
      "type": "value",
      "value": "title"
    }
  },
  {
    "type": "put",
    "value": {
      "type": "replace",
      "value": {
        "type": "get",
        "source": "content"
      },
      "replace": [
        {
          "to": "<Add text here that will replace text below>",
          "from": "<Add text to remove here>"
        }
      ]
    },
    "target": {
      "type": "value",
      "value": "content"
    }
  },
  {
    "type": "put",
    "value": {
      "type": "get",
      "source": "excerpt"
    },
    "target": {
      "type": "value",
      "value": "excerpt"
    }
  }
]
```


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.ebbot.ai/ebbot-docs/core-capabilities/ebbotgpt/ebbotgpt-knowledge/knowledge-pre-processing/advanced-transformer-settings.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.