Vector Index Extensions

The MatrixOne Python SDK provides advanced vector indexing and similarity search capabilities through SQLAlchemy extensions.

VectorIndex

class matrixone.sqlalchemy_ext.vector_index.VectorIndex(name: str, column: str | Column, index_type: str = 'ivfflat', lists: int | None = None, op_type: str = 'vector_l2_ops', m: int | None = None, ef_construction: int | None = None, ef_search: int | None = None, **kwargs)[source]

Bases: Index

SQLAlchemy Index for vector columns with MatrixOne-specific syntax.

This class provides a generic interface for creating vector indexes with various algorithms and operation types. It supports both IVF (Inverted File) and HNSW (Hierarchical Navigable Small World) indexing algorithms.

Key Features:

  • Support for multiple vector indexing algorithms (IVF, HNSW)

  • Configurable operation types (L2 distance, cosine similarity, inner product)

  • Automatic SQL generation for index creation and management

  • Integration with MatrixOne’s vector search capabilities

  • Support for both class methods and instance methods

Supported Index Types: - IVF (Inverted File): Good for large datasets, requires training - HNSW: Good for high-dimensional vectors, no training required

Supported Operation Types: - VECTOR_L2_OPS: L2 (Euclidean) distance - VECTOR_COSINE_OPS: Cosine similarity - VECTOR_INNER_PRODUCT_OPS: Inner product similarity

Usage Examples:

# Create IVF index
index = VectorIndex(
    name='vec_idx_ivf',
    column='embedding',
    index_type=VectorIndexType.IVFFLAT,
    lists=100,
    op_type=VectorOpType.VECTOR_L2_OPS
)

# Create HNSW index
index = VectorIndex(
    name='vec_idx_hnsw',
    column='embedding',
    index_type=VectorIndexType.HNSW,
    m=16,
    ef_construction=200,
    op_type=VectorOpType.VECTOR_COSINE_OPS
)

Note: This is the legacy generic class. For better type safety and specific algorithm features, consider using IVFVectorIndex or HnswVectorIndex instead.

__init__(name: str, column: str | Column, index_type: str = 'ivfflat', lists: int | None = None, op_type: str = 'vector_l2_ops', m: int | None = None, ef_construction: int | None = None, ef_search: int | None = None, **kwargs)[source]

Initialize VectorIndex.

Args:

name: Index name
column: Vector column to index
index_type: Type of vector index (ivfflat, hnsw, etc.)
lists: Number of lists for IVFFLAT (optional)
op_type: Vector operation type
m: Number of bi-directional links for HNSW (optional)
ef_construction: Size of dynamic candidate list for HNSW construction (optional)
ef_search: Size of dynamic candidate list for HNSW search (optional)
**kwargs: Additional index parameters
create_sql(table_name: str) → str[source]

Generate CREATE INDEX SQL for the given table name.

drop_sql(table_name: str) → str[source]

Generate DROP INDEX SQL for the given table name.

classmethod create_index(engine, table_name: str, name: str, column: str | Column, index_type: str = 'ivfflat', lists: int | None = None, op_type: str = 'vector_l2_ops', m: int | None = None, ef_construction: int | None = None, ef_search: int | None = None, **kwargs) → bool[source]

Create a vector index using ORM-style method.

Args:

engine: SQLAlchemy engine
table_name: Name of the table
name: Name of the index
column: Vector column to index
index_type: Type of vector index (ivfflat, hnsw, etc.)
lists: Number of lists for IVFFLAT (optional)
op_type: Vector operation type
m: Number of bi-directional links for HNSW (optional)
ef_construction: Size of dynamic candidate list for HNSW construction (optional)
ef_search: Size of dynamic candidate list for HNSW search (optional)
**kwargs: Additional index parameters

Returns:

bool: True if successful, False otherwise
classmethod drop_index(engine, table_name: str, name: str) → bool[source]

Drop a vector index using ORM-style method.

Args:

engine: SQLAlchemy engine
table_name: Name of the table
name: Name of the index to drop

Returns:

bool: True if successful, False otherwise
create(engine, table_name: str) → bool[source]

Create this vector index using ORM-style method.

Args:

engine: SQLAlchemy engine
table_name: Name of the table

Returns:

bool: True if successful, False otherwise
drop(engine, table_name: str) → bool[source]

Drop this vector index using ORM-style method.

Args:

engine: SQLAlchemy engine
table_name: Name of the table

Returns:

bool: True if successful, False otherwise

HNSWConfig

class matrixone.sqlalchemy_ext.hnsw_config.HNSWConfig(engine: Engine)[source]

Bases: object

Configuration manager for HNSW indexing in MatrixOne.

Provides methods to enable/disable HNSW indexing and manage HNSW-specific parameters.

__init__(engine: Engine)[source]

Initialize HNSWConfig.

Parameters:

engine – SQLAlchemy engine instance

enable_hnsw_indexing(connection: Connection | None = None) → bool[source]

Enable HNSW indexing in MatrixOne.

Parameters:

connection – Optional existing database connection

Returns:

True if successful, False otherwise

Return type:

bool

disable_hnsw_indexing(connection: Connection | None = None) → bool[source]

Disable HNSW indexing in MatrixOne.

Parameters:

connection – Optional existing database connection

Returns:

True if successful, False otherwise

Return type:

bool

get_hnsw_status(connection: Connection | None = None) → bool | None[source]

Get the current HNSW indexing status.

Parameters:

connection – Optional existing database connection

Returns:

True if HNSW indexing is enabled, False if disabled, None if error

Return type:

bool

IVFConfig

class matrixone.sqlalchemy_ext.ivf_config.IVFConfig(engine: Engine)[source]

Bases: object

Configuration manager for IVF indexing in MatrixOne.

This class provides utilities to enable/disable IVF indexing and configure search parameters.

__init__(engine: Engine)[source]

Initialize IVF configuration manager.

Parameters:

engine – SQLAlchemy engine connected to MatrixOne

enable_ivf_indexing() → bool[source]

Enable IVF indexing in MatrixOne.

Returns:

True if successful, False otherwise

disable_ivf_indexing() → bool[source]

Disable IVF indexing in MatrixOne.

Returns:

True if successful, False otherwise

set_probe_limit(limit: int) → bool[source]

Set the probe limit for IVF index search.

Parameters:

limit – Number of probes to use during search

Returns:

True if successful, False otherwise

get_ivf_status() → Dict[str, Any][source]

Get current IVF configuration status.

Returns:

Dictionary with current IVF settings

configure_ivf(enabled: bool = True, probe_limit: int | None = None) → bool[source]

Configure IVF indexing with multiple parameters.

Parameters:

  • enabled – Whether to enable IVF indexing

  • probe_limit – Probe limit for search (optional)

Returns:

True if all operations successful, False otherwise

is_ivf_supported() → bool[source]

Check if IVF indexing is supported in the current MatrixOne instance.

Returns:

True if IVF is supported, False otherwise

Usage Examples

Basic Vector Index Creation

from matrixone import Client
from matrixone.sqlalchemy_ext import VectorIndex, HNSWConfig

client = Client()
client.connect(host='localhost', port=6001, user='root', password='111', database='test')

# Create a table with vector column
client.execute("""
    CREATE TABLE embeddings (
        id INT PRIMARY KEY,
        text VARCHAR(200),
        embedding VECTOR(384)
    )
""")

# Create HNSW vector index
config = HNSWConfig(m=16, ef_construction=200)
client.vector_ops.create(
    table_name='embeddings',
    name='idx_embedding',
    column='embedding',
    algorithm='hnsw',
    config=config
)

# Search for similar vectors
result = client.execute("""
    SELECT id, text, embedding <-> '[0.1, 0.2, ...]' as distance
    FROM embeddings
    ORDER BY embedding <-> '[0.1, 0.2, ...]'
    LIMIT 10
""")

Advanced Vector Operations

from matrixone.sqlalchemy_ext import VectorIndex, IVFConfig

# Create IVF vector index
config = IVFConfig(nlist=100, nprobe=10)
client.vector_ops.create(
    table_name='embeddings',
    name='idx_ivf',
    column='embedding',
    algorithm='ivf',
    config=config
)

# Batch insert vectors
vectors = [
    (1, 'Document 1', [0.1, 0.2, 0.3, ...]),
    (2, 'Document 2', [0.4, 0.5, 0.6, ...]),
    (3, 'Document 3', [0.7, 0.8, 0.9, ...])
]

for vector_data in vectors:
    client.execute(
        "INSERT INTO embeddings (id, text, embedding) VALUES (%s, %s, %s)",
        vector_data
    )

# Search with different distance metrics
result = client.execute("""
    SELECT id, text,
           embedding <#> '[0.1, 0.2, ...]' as cosine_distance,
           embedding <-> '[0.1, 0.2, ...]' as l2_distance
    FROM embeddings
    ORDER BY embedding <-> '[0.1, 0.2, ...]'
    LIMIT 5
""")

Async Vector Operations

import asyncio
from matrixone import AsyncClient

async def vector_async_example():
    async_client = AsyncClient()
    await async_client.connect(
        host='localhost',
        port=6001,
        user='root',
        password='111',
        database='test'
    )

    # Create vector index asynchronously
    config = HNSWConfig(m=16, ef_construction=200)
    await async_client.vector_index.create(
        table_name='embeddings',
        name='idx_async',
        column='embedding',
        algorithm='hnsw',
        config=config
    )

    # Search asynchronously
    result = await async_client.execute("""
        SELECT id, text FROM embeddings
        ORDER BY embedding <-> '[0.1, 0.2, ...]'
        LIMIT 10
    """)

    for row in result:
        print(f"Found: {row[1]}")

    await async_client.disconnect()

asyncio.run(vector_async_example())

Vector Index Management

# List all vector indexes
indexes = client.vector_ops.list()
for idx in indexes:
    print(f"Index: {idx.name}, Table: {idx.table_name}")

# Get index information
info = client.vector_ops.get('idx_embedding')
print(f"Index algorithm: {info.algorithm}")
print(f"Index config: {info.config}")

# Drop vector index
client.vector_ops.drop('idx_embedding')

# Rebuild index
client.vector_ops.rebuild('idx_embedding')