Topics

#django #django-application #django-packages #django-redis #django-search #django-search-engine #python #python-redis #python3 #redis #redis-django #redis-python #redis-search #redis-search-django #redisearch #redisjson

Language breakdown

Python 95.0%

HTML 5.0%

▸ Files click to expand

README

redis-search-django

About

A Django package that provides auto indexing and searching capabilities for Django model instances using RediSearch.

Features

Management Command to create, update and populate the RediSearch Index.
Auto Index on Model object Create, Update and Delete.
Auto Index on Related Model object Add, Update, Remove and Delete.
Easy to create Document classes (Uses Django Model Form Class like structure).
Index nested models (e.g: OneToOneField, ForeignKey and ManyToManyField).
Search documents using redis-om.
Search Result Pagination.
Search Result Sorting.
RediSearch Result to Django QuerySet.
Faceted Search.

Requirements

Python: 3.7, 3.8, 3.9, 3.10
Django: 3.2, 4.0, 4.1
redis-om: >= 0.0.27

Redis

Downloading Redis

The latest version of Redis is available from Redis.io. You can also install Redis with your operating system's package manager.

RediSearch and RedisJSON

redis-search-django relies on the RediSearch and RedisJSON Redis modules to support rich queries and embedded models. You need these Redis modules to use redis-search-django.

The easiest way to run these Redis modules during local development is to use the redis-stack Docker image.

Docker Compose

There is a docker-compose.yaml file provided in the project's root directory. This file will run Redis with RedisJSON and RediSearch modules during development.

Run the following command to start the Redis container:

docker compose up -d

Example Project

There is an example project available at Example Project.

Documentation

Installation

pip install redis-search-django

Then add redissearchdjango to your INSTALLED_APPS:

INSTALLED_APPS = [
    ...
    'redissearchdjango',
]

Usage

Document Types

There are 3 types of documents class available:

JsonDocument: This uses RedisJSON to store the document. If you want to use Embedded Documents (Required For OneToOneField, ForeignKey and ManyToManyField) then use JsonDocument.
EmbeddedJsonDocument: If the document will be embedded inside another document class then use this. Embedded Json Documents are used for OneToOneField, ForeignKey and ManyToManyField or any types of nested documents.
HashDocument: This uses RedisHash to store the documents. It can not be used for nested documents.

Creating Document Classes

You need to inherit from The Base Document Classes mentioned above to build a document class.

Simple Example

1. For Django Model:

# models.py
from django.db import models
class Category(models.Model):     name = models.CharField(max_length=30)     slug = models.SlugField(max_length=30)
def str(self) -> str:         return self.name

2. You can create a document class like this:

Note: Document classes must be stored in documents.py file.

# documents.py
from redissearchdjango.documents import JsonDocument
from .models import Category
class CategoryDocument(JsonDocument):     class Django:         model = Category         fields = ["name", "slug"]

3. Run Index Django Management Command to create the index on Redis:

python manage.py index

Note: This will also populate the index with existing data from the database

Now category objects will be indexed on create/update/delete.

More Complex Example

1. For Django Models:

# models.py
from django.db import models
class Tag(models.Model):     name = models.CharField(max_length=30)
def str(self) -> str:         return self.name
class Vendor(models.Model):     name = models.CharField(max_length=30)     email = models.EmailField()     establishment_date = models.DateField()
def str(self) -> str:         return self.name
class Product(models.Model):     name = models.CharField(max_length=256)     description = models.TextField(blank=True)     vendor = models.OneToOneField(Vendor, on_delete=models.CASCADE)     tags = models.ManyToManyField(Tag, blank=True)     price = models.DecimalField(maxdigits=6, decimalplaces=2)
def str(self) -> str:         return self.name

2. You can create a document classes like this:

Note: Document classes must be stored in documents.py file.

# documents.py
from typing import List
from django.db import models from redis_om import Field
from redissearchdjango.documents import EmbeddedJsonDocument, JsonDocument
from .models import Product, Tag, Vendor
class TagDocument(EmbeddedJsonDocument):     customfield: str = Field(index=True, fulltext_search=True)
class Django:         model = Tag         # Model Fields         fields = ["name"]
@classmethod     def preparecustomfield(cls, obj):         return "CUSTOM FIELD VALUE"
class VendorDocument(EmbeddedJsonDocument):     class Django:         model = Vendor         # Model Fields         fields = ["name", "establishment_date"]
class ProductDocument(JsonDocument):     # OnetoOneField, with null=False     vendor: VendorDocument     # ManyToManyField     tags: List[TagDocument]
class Django:         model = Product         # Model Fields         fields = ["name", "description", "price"]         # Related Model Options         related_models = {             Vendor: {                 "related_name": "product",                 "many": False,             },             Tag: {                 "relatedname": "productset",                 "many": True,             },         }
@classmethod     def get_queryset(cls) -> models.QuerySet:         """Override Queryset to filter out available products."""         return super().get_queryset().filter(available=True)
@classmethod     def prepare_name(cls, obj):         """Use this to update field value."""         return obj.name.upper()

Note:

You can not inherit from HashDocument for documents that include nested fields.
You need to inherit from EmbeddedJsonDocument for document classes that will be embedded inside another document class.
You need to explicitly add OneToOneField, ForeignKey or ManyToManyField (e.g: tags: List[TagDocument]) with an embedded document class if you want to index them.

you can not add it in the Django.fields option.

For relatedmodels option, you need to specify the fields relatedname and if it is a ManyToManyField or a ForeignKey Field then specify "many": True.
related_models will be used when a related object is saved that contributes to the document.
You can define prepare{fieldname} method to update the value of a field before indexing.
If it is a custom field (not a model field) you must define a prepare{fieldname} method that returns the value of the field.
You can override get_queryset method to provide more filtering. This will be used while indexing a queryset.
Field names must match model field names or define a prepare{fieldname} method.

3. Run Index Django Management Command to create the index on Redis:

python manage.py index

Note: This will also populate the index with existing data from the database

Management Command

This package comes with index management command that can be used to index all the model instances to Redis index if it has a Document class defined.

Note: Make sure that Redis is running before running the command.

Run the following command to index all models that have Document classes defined:

python manage.py index

You can use --migrate-only option to only update the index schema.

python manage.py index --migrate-only

You can use --models to specify which models to index (models must have a Document class defined to be indexed).

python manage.py index --models appname.ModelName appname2.ModelName2

Views

You can use the redissearchdjango.mixin.RediSearchListViewMixin with a Django Generic View to search for documents. RediSearchPaginator which helps paginate ReadiSearch results is also added to this mixin.

Example

# views.py
from django.utils.functional import cached_property from django.views.generic import ListView from redis.commands.search import reducers
from redissearchdjango.mixins import RediSearchListViewMixin
from .documents import ProductDocument from .models import Product
class SearchView(RediSearchListViewMixin, ListView):     paginate_by = 20     model = Product     template_name = "core/search.html"     document_class = ProductDocument
@cached_property     def searchqueryexpression(self):         query = self.request.GET.get("query")         query_expression = None
if query:             query_expression = (                 self.document_class.name % query                 | self.document_class.description % query             )
return query_expression
@cached_property     def sort_by(self):         return self.request.GET.get("sort")
def facets(self):         if self.searchqueryexpression:             request = self.documentclass.buildaggregate_request(                 self.searchqueryexpression             )         else:             request = self.documentclass.buildaggregate_request()
result = self.document_class.aggregate(             request.group_by(                 ["@tags_name"],                 reducers.count().alias("count"),             )         )         return result

Search

This package uses redis-om to search for documents.

Example

from .documents import ProductDocument
categories = ["category1", "category2"] tags = ["tag1", "tag2"]
Search For Products That Match The Search Query (name or description)
query_expression = (
    ProductDocument.name % "Some search query"
    | ProductDocument.description % "Some search query"
)
Search For Products That Match The Price Range
query_expression = (
    ProductDocument.price >= float(10) & ProductDocument.price <= float(100)
)
Search for Products that include following Categories
query_expression = ProductDocument.category.name << ["category1", "category2"]
Search for Products that include following Tags
query_expression = ProductDocument.tags.name << ["tag1", "tag2"]
Query expression can be passed on the find method
result = ProductDocument.find(queryexpression).sortby("-price").execute()

For more details checkout redis-om docs

RediSearch Aggregation / Faceted Search

redis-om does not support faceted search (RediSearch Aggregation). So this package uses redis-py to do faceted search.

Example

from redis.commands.search import reducers
from .documents import ProductDocument
query_expression = (     ProductDocument.name % "Some search query"     | ProductDocument.description % "Some search query" )
First we need to build the aggregation request
request1 = ProductDocument.buildaggregaterequest(query_expression)
request2 = ProductDocument.buildaggregaterequest(query_expression)
Get the number of products for each category
ProductDocument.aggregate(
    request1.group_by(
        ["@category_name"],
        reducers.count().alias("count"),
    )
)
>> [{"categoryname": "Shoes", "count": "112"}, {"categoryname": "Cloths", "count": "200"}]
Get the number of products for each tag
ProductDocument.aggregate(
    request2.group_by(
        ["@tags_name"],
        reducers.count().alias("count"),
    )
)
>> [{"tagsname": "Blue", "count": "14"}, {"tagsname": "Small", "count": "57"}]

For more details checkout redis-py docs and RediSearch Aggregation docs

Settings

Environment Variables

REDISOMURL (Default: redis://localhost:6379): This environment variable follows the redis-py URL format. If you are using external redis server

You need to set this variable with the URL of the redis server following this pattern: redis://[[username]:[password]]@[host]:[post]/[database number]

Example: redis://redis_user:password@some.other.part.cloud.redislabs.com:6379/0

For more details checkout redis-om docs

Django Document Options

You can add these options on the Django class of each Document class:

# documents.py
from redissearchdjango.documents import JsonDocument
from .models import Category, Product, Tag, Vendor
class ProductDocument(JsonDocument):     class Django:         model = Product         fields = ["name", "description", "price", "created_at"]         selectrelatedfields = ["vendor", "category"]         prefetchrelatedfields = ["tags"]         auto_index = True         related_models = {             Vendor: {                 "related_name": "product",                 "many": False,             },             Category: {                 "relatedname": "productset",                 "many": True,             },             Tag: {                 "relatedname": "productset",                 "many": True,             },         }

model (Required): Django Model class to index.
auto_index (Default: True, Optional): If True, the model instances will be indexed on create/update/delete.
fields (Default: [], Optional): List of model fields to index. (Do not add OneToOneField, ForeignKey or ManyToManyField here. These need to be explicitly added to the Document class using EmbeddedJsonDocument.)
selectrelatedfields (Default: [], Optional): List of fields to use on queryset.select_related().
prefetchrelatedfields (Default: [], Optional): List of fields to use on queryset.prefetch_related().
related_models (Default: {}, Optional): Dictionary of related models.

You need to specify the fields related_name and if it is a ManyToManyField or a ForeignKey Field then specify "many": True. These are used to update the document data if any of the related model instances are updated. related_models will be used when a related object is saved/added/removed/deleted that contributes to the document.

For redis-om specific options checkout redis-om docs