locality sensitive hashing (LSHASH) for Python3
LSHash ======
:Version: 0.0.9 :Python: 3.11.5
A fast Python implementation of locality sensitive hashing with persistance support.
Based on original source code https://github.com/kayzhu/LSHash
Highlights ==========
- Python3 support
- Load & save hash tables to local disk
- Fast hash calculation for large amount of high dimensional data through the use of
numpyarrays. - Built-in support for persistency through Redis.
- Multiple hash indexes support.
- Built-in support for common distance/objective functions for ranking outputs.
LSHash depends on the following libraries:
- numpy
- bitarray (if hamming distance is used as distance function)
Optional
- redis (if persistency through Redis is needed)
To install from sources:
.. code-block:: bash
$ git clone https://github.com/loretoparisi/lshash.git $ python setup.py install To install from PyPI:
.. code-block:: bash
$ pip install lshashpy3 $ python -c "import lshashpy3 as lshash; print(lshash.version);"
Quickstart ========== To create 6-bit hashes for input data of 8 dimensions:
.. code-block:: python
# create 6-bit hashes for input data of 8 dimensions: lsh = LSHash(6, 8) # index vector lsh.index([2,3,4,5,6,7,8,9])
# index vector and extra data lsh.index([10,12,99,1,5,31,2,3], extra_data="vec1") lsh.index([10,11,94,1,4,31,2,3], extra_data="vec2")
# query a data point top_n = 1 nn = lsh.query([1,2,3,4,5,6,7,7], numresults=topn, distance_func="euclidean") print(nn)
# unpack vector, extra data and vectorial distance top_n = 3 nn = lsh.query([10,12,99,1,5,30,1,1], numresults=topn, distance_func="euclidean") for ((vec,extra_data),distance) in nn: print(vec, extra_data, distance) To save hash table to disk:
.. code-block:: python
lsh = LSHash(hashsize=k, inputdim=d, num_hashtables=L, storage_config={ 'dict': None }, matrices_filename='weights.npz', hashtable_filename='hash.npz', overwrite=True)
lsh.index([10,12,99,1,5,31,2,3], extra_data="vec1") lsh.index([10,11,94,1,4,31,2,3], extra_data="vec2") lsh.save()
To load hash table from disk and perform a query:
.. code-block:: python
lsh = LSHash(hashsize=k, inputdim=d, num_hashtables=L, storage_config={ 'dict': None }, matrices_filename='weights.npz', hashtable_filename='hash.npz', overwrite=False)
top_n = 3 nn = lsh.query([10,12,99,1,5,30,1,1], numresults=topn, distance_func="euclidean") print(nn)
New Feature: Multiprocessing Support ===================================
The library now supports indexing multiple items in parallel using the
index_batch method. This feature leverages Python's multiprocessing module to speed up the indexing process for large datasets.
Example: Using Multiprocessing for Batch Indexing
To index multiple items in parallel:
.. code-block:: python
from lshashpy3 import LSHash
# Create an LSHash instance lsh = LSHash(hashsize=6, inputdim=8, num_hashtables=5)
# Define input points and optional extra data input_points = [ [2, 3, 4, 5, 6, 7, 8, 9], [10, 12, 99, 1, 5, 31, 2, 3], [10, 11, 94, 1, 4, 31, 2, 3], [1, 2, 3, 4, 5, 6, 7, 7], [10, 12, 99, 1, 5, 30, 1, 1] ] extradatalist = ["vec1", "vec2", "vec3", "vec4", "vec5"]
# Index the points in parallel lsh.indexbatch(inputpoints, extradatalist)
# Verify the indexed data for point, extradata in zip(inputpoints, extradatalist): hashes = lsh.get_hashes(point) print(f"Point: {point}, Extra Data: {extra_data}, Hashes: {hashes}")
API ==============
- To initialize a
LSHash instance:
.. code-block:: python
k = 6 # hash size L = 5 # number of tables d = 8 # Dimension of Feature vector LSHash(hashsize=k, inputdim=d, num_hashtables=L, storage_config={ 'dict': None }, matrices_filename='weights.npz', hashtable_filename='hash.npz', overwrite=True)
parameters:
hash_size: The length of the resulting binary hash. input_dim: The dimension of the input vector. num_hashtables = 1: (optional) The number of hash tables used for multiple lookups. storage = None: (optional) Specify the name of the storage to be used for the index storage. Options include "redis". matrices_filename = None: (optional) Specify the path to the .npz file random matrices are stored or to be stored if the file does not exist yet hashtable_filename = None: (optional) Specify the path to the .npz file hash table are stored or to be stored if the file does not exist yet overwrite = False: (optional) Whether to overwrite the matrices file if it already exist
- To index a data point of a given
LSHash instance, e.g., lsh:
.. code-block:: python
lsh.index(inputpoint, extradata=None):
parameters:
input_point: The input data point is an array or tuple of numbers of input_dim. extra_data = None: (optional) Extra data to be added along with the input_point.
- To query a data point against a given
LSHash instance, e.g., lsh:
.. code-block:: python
lsh.query(querypoint, numresults=None, distance_func="euclidean"):
parameters:
query_point: The query data point is an array or tuple of numbers of input_dim. num_results = None: (optional) The number of query results to return in ranked order. By default all results will be returned. distance_func = "euclidean"`: (optional) Distance function to use to rank the candidates. By default "euclidean" distance function will be used. Distance function can be "euclidean", "trueeuclidean", "centredeuclidean", "cosine", "l1norm".
- To save the hash table currently indexed:
lsh.save():
