Supercharge your search with

Typesense

                | ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄|
                   Open Source Alternative  
                     to Solr and Elastic        
                |________________|
                          \(•◡•)/
                           \   /
                            ---
                            | |
                            |_|_

Actual ad: x.com/typesense/status/1702152406924980670

About @null (╯°□°)╯

  • Senior Developer, Trade Me, Wellington NZ
  • Web development since 1996
  • 25+ years PHP & search
  • Loves corgis

TL;DR 🤯

Typesense is a search engine for your website that does clever things much faster than anything else you've seen

TL;DR: DEMO 🤯

typesense.org

What you can do 🦸

🔥 Instantaneous sub-ms response times
✅ Faceting
✅ Sorting
✅ Curation
✅ Synonyms
✅ Scoped search
✅ Geospatial
✅ Automatic schema detection
✅ Analytics
✅ Document text extraction
✅ Multi-language support
✅ Federated search
✅ Semantic & natural search
✅ Decentralised management dashboard
✅ AI / Machine learning with GPU support

How does it work?

  • Search as you type
  • Peerable and distributed for redundancy
  • Performance. Cost. Ease-of use
  • Self-hosted, or cloud-based. Your choice

About Typesense

  • Created in 2016 by Jason Bosco and Kishore Nallan
  • Alternatives were hard to learn or expensive (or both)
  • Written in C++

Typesense ❤️ FLOSS

  • Free to use, GPLv3
  • SaaS hosting available
  • "CDN for extremely fast search"

Performance 🏍️

Amount RAM Time
2.4M Addresses 600MB ~5 minutes
2.2M Recipes 900MB 3.6 minutes
28M Books 14GB 76 minutes
250 concurrent search queries per second

Curated search results 🪜

  • Promote one product over another
  • Change order of specific search results
  • EXAMPLE: Featured pages during a sale

Curated search results 🪜

  • Exclude unsupported products
  • Demote pages in specific search results
  • EXAMPLE: Products out of stock

Synonyms 🔄

  • "Silverstripe", "Drupal", and "Laravel" <=> "CMS".
  • "smart phone" => "android", "iphone".
  • Curations always precede synonyms

Built-in analytics 🧐

  • Popular queries
  • "No-hit" queries
  • Track how often a particular result is clicked on
  • Integrates with Matomo and Google

But wait, there's more!

Typesense Dashboard

  • Decentralised and offline, or locally installed
  • Connects to any installation on Earth with CORS
  • Search collections, update schema, server metrics

🔍 Federated search 🔎

  • Concurrent searches in a single HTTP request
  • Site-wide search with different result sets

Scoped search 🛡️

  • Safely offer authenticated privileged search
  • Cryptographically signed with unique user attributes
  • One collection, one application, many keys
  • A different key give different results, or none

Geospatial searches 🌐

  • Radius, or within a polygon, or outside of them
  • GIS applications, such as council plans, Homes.co.nz, etc

Language support 🗣️

  • Full unicode support, including te reo Māori
  • non-Latin alphabets, (Cyrillic, Chinese, etc)
  • Snowball Stemmer
  • Stop-words for different languages

Text extraction with Tika 📖

  • Works perfectly with silverstripe/textextraction
  • Extract text from documents
  • Containerized, no dependency on Solr

AI and machine-learning 🤖

  • Adds context to results.
  • Index-time calculations.
  • LLM reads and summarises.

AI and machine-learning 🤖

  • Image search.
  • Audio / video transcriptions.
  • Huge win for accessibility

...My webserver needs a GPU? 🤔

  • No! Typesense can use your CPU instead
  • If you have a GPU available, it can use it
  • Compatible with OpenAI, Google, and others
  • Turn it off when you're done with it

Key takeaways

  • Easy to deploy
  • Save on development time
  • Performance and user experience
  • Cost savings

FindXKCD

Paint your website with Resene

The collection

							```js
{
  "name": "Colours",
  "fields": [
    { "name": ".*", "type": "auto", "optional": true },
    { "name": "Charts", "type": "string[]", "facet": true, "sort": false },
    { "name": "Category", "type": "string", "facet": true, "sort": false },
    { 
	  "name": "rgb_vector",
	  "type": "float[]",
	  "facet": false,
	  "sort": false,
	  "num_dim": 3,
	  "vec_dist": "cosine" 
	}
  ]
}
```
					

Instant Search

Fetch API makes the vector search

```
const response = await fetch(
	`${typesenseUri}/collections/${searchCollection}/documents/search
		?q=*
		&vector_query=${encodeURIComponent(vectorQuery)}
		&exclude_fields=rgb_vector
		&sort_by=Blue:desc
	`,
	{
		headers: {
			'X-TYPESENSE-API-KEY': TYPESENSE_CONFIG.apiKey
		}
	}
);

const data = await response.json();
```
						

Paint your website slide deck with Resene

Blurple
#4854D3
Fugitive
#C3423B
Smitten
#C84086

NZ Address Lookup

The collection schema

```
{
	name: Addresses
	fields: [
		{ name: Region, type: string, facet: true, optional: true }
		{ name: Council, type: string, facet: true, optional: true }
		{ name: City, type: string, facet: true, optional: true }
		{ name: Suburb, type: string, facet: true, optional: true }
		{ name: FullAddress, type: string}
		{ name: Location, type: geopoint}
	]
}
```
						

A PHP importer

							```php

    public function run($request) {
        $this->preprocess();
        $this->importSuburbs();
        $this->deleteTypesenseCollection();
        $this->buildTypesenseCollection();
    }

    public function preprocess()
    {
        $councilsDB = DB::query("SELECT ID, Name FROM NZTerritorialAuthority");
        foreach($councilsDB as $council) {
            $this->councils[$council['ID']] = $council['Name'];
        }

        $citiesDB = DB::query("SELECT ID, Name FROM NZTownCity");
        foreach($citiesDB as $city) {
            $this->cities[$city['ID']] = $city['Name'];
        }

        $suburbsDB = DB::query("SELECT ID, Name FROM NZSuburbLocality");
        foreach($suburbsDB as $suburb) {
            $this->suburbs[$suburb['ID']] = $suburb['Name'];
        }
    }

    public function deleteTypesenseCollection()
    {
        try {
            $client = Typesense::client();
            $client->collections['suburbs']->delete();
        } catch (Exception $e) {}
    }

    public function buildTypesenseCollection()
    {
        try {
            $client = Typesense::client();
            $client->collections->create([
                'name' => 'suburbs',
                'enable_nested_fields' => true,
                'fields' => [
                    [ 'name' => 'council.name', 'type' => 'string' ],
                    [ 'name' => 'city.name', 'type' => 'string' ],
                    [ 'name' => 'suburb.name', 'type' => 'string' ],
                    [ 'name' => 'polyline', 'type' => 'string' ],
                ]
            ]);

            $client->collections['suburbs']->documents->import($this->documents, ['action' => 'emplace']);

        } catch (Exception $e) {}
    }

    public function importSuburbs()
    {

        $suburbsDB = DB::query("SELECT ID, CouncilID, CityID, SuburbID, Polyline FROM NZSuburb");
        foreach($suburbsDB as $suburb) {
            $councilName = $this->councils[$suburb['CouncilID']];
            $cityName = $this->cities[$suburb['CityID']];
            $suburbName = $this->suburbs[$suburb['SuburbID']];
            $polyline = $suburb['Polyline'];

            if($councilName && $cityName && $suburbName && $polyline) {
                $this->documents[] = [
                    'id' => (string) $suburb['ID'],
                    'council.name' => $councilName,
                    'city.name' => $cityName,
                    'suburb.name' => $suburbName,
                    'polyline' => $polyline,
                ];

            }
        }
    }
```							
						

Javascript Magic

```js
// Listen for refinement changes to display suburb polyline
//search is our InstantSearch client
search.on('render', function() {
	//...
	// Display the suburb polyline
	if (cityName && councilName) {
		displaySuburbPolyline(suburbName, cityName, councilName);
	}
}
async function displaySuburbPolyline(...) {
	//clear layers, bail if any parameters empty
	try {
		const searchParameters = {
			q: '*', query_by: 'suburb.name', filter_by: `suburb.name:=${suburbName} && city.name:=${cityName} && council.name:=${councilName}`, per_page: 1
		};

		const results = await typesenseClient.collections('suburbs').documents().search(searchParameters);
		if (results.hits && results.hits.length > 0) {
			//decode polyline with mapbox, add it to map
		}
	}
}
```
						

Scoped Search

Schema

```js
{
      name: SharedDocuments,
      enable_nested_fields: true,
      fields: [
	      { name: Title, type: string, sort: true },
	      { name: Content, type: string, optional: true },
	      { name: Link, type: string, index: false, optional: true},
	      { name: Owner.ID, type: int32[], index: true, optional: false, sort: false},
	      { name: Owner.Name, type: string[], facet: true, index: true, optional: false, sort: false},
	      { name: Owner.Email, type: string[], facet: false, index: false, optional: false, sort: false}
	  ]
}
```

Application

API Key

```php
class ScopedSearchPageController extends TypesenseSearchPageController
{
    public function init()
    {
        parent::init();
        if(!($member = Security::getCurrentUser())) {
            return;
        }
        $parentKey = $this->data()->SearchAPIKey;
        if($member && !$member->TypesenseAPISearchKey) {
            $client = Typesense::client();
            $expiry = strtotime("+24 hours");

            if(!$member->TypesenseAPISearchKey || $member->TypesenseAPISearchKeyExpires > DBDatetime::now()) {
                $scopedKey = $client->keys->generateScopedSearchKey($parentKey, [
                    'filter_by' => 'Owner.ID:'.$member->ID,
                    'expires_at' => $expiry
                ]);
                if($scopedKey) {
                    $member->TypesenseAPISearchKey = $scopedKey;
                    $member->TypesenseAPISearchKeyExpires = $expiry;
                    $member->write();
                }
            }

        }
    }

}
```
						

Audiobook search

  • OpenAI Whisper produces a transcript
  • Parsed line-by-line

AWESOME

Batteries included! 🔋

Silverstripe Typesense module

  • Configure with YML or inside of the CMS
  • Syncs remote or local, auto-update on save
  • Synonyms and translations
  • Under development, contributions welcome!

More info about Typesense

  • typesense.org/
  • cloud.typesense.org/
  • github.com/typesense/typesense
  • codeberg.org/0x/silverstripe-typesense
 

Credits!

 
  • Rosemary
  • Simon E
  • David M
  • Trade Me
 
(╯°□°)╯

Thank you!

 
 
 
  • Slack: @null
  • github.com/elliot-sawyer
  • codeberg.org/0x
  • linkedin.com/in/elliot-sawyer/
  • elliot@cashware.nz
  • sawyer.nz/demos
 
 
 
Back to main site