Comprehensive guides and references for the OpenFrame platform
The Data Pinot Repositories And Models module provides the analytical data access layer for Apache Pinot within the OpenFrame platform. It is responsible for querying high-volume, time-series, and event-driven datasets such as:
Unlike MongoDB repositories (which handle transactional and document-based storage), this module focuses on low-latency analytical queries optimized for filtering, aggregation, and search over large datasets indexed in Pinot.
This module contains:
PinotEventEntity – base model placeholder for Pinot event representations PinotClientDeviceRepository – device aggregation and filter repository PinotClientLogRepository – log search and analytical repositoryApache Pinot is used as the real-time OLAP engine in the OpenFrame data platform. Events are ingested via Kafka, enriched by stream processors, and stored in Pinot tables optimized for fast filtering and aggregation.
flowchart LR
Kafka["Kafka Topics"] --> Stream["Stream Service"]
Stream --> Pinot["Apache Pinot"]
Pinot --> DeviceRepo["PinotClientDeviceRepository"]
Pinot --> LogRepo["PinotClientLogRepository"]
DeviceRepo --> ApiLayer["API / GraphQL Layer"]
LogRepo --> ApiLayer
| Layer | Responsibility |
|---|---|
| Kafka | Event ingestion |
| Stream Service | Event normalization & enrichment |
| Apache Pinot | Analytical storage & indexing |
| Data Pinot Repositories And Models | Query construction & execution |
| API / GraphQL | External data exposure |
Package: com.openframe.data.model.pinot
PinotEventEntity is a model placeholder representing a Pinot-backed event entity. While currently empty, it establishes a domain anchor for Pinot-based event representations.
Package: com.openframe.data.repository.pinot
PinotClientDeviceRepository provides aggregation and filtering queries over the devices Pinot table.
It is optimized for:
@Bean qualifier: "pinotBrokerConnection"
Property: pinot.tables.devices.name (default: "devices")
GROUP BY DELETED device statusflowchart TD
Input["Filter Input Lists"] --> BuildWhere["Build WHERE Clause"]
BuildWhere --> ComposeQuery["Compose SELECT / GROUP BY"]
ComposeQuery --> Execute["pinotConnection.execute()"]
Execute --> MapResults["Map ResultSet to Map<String,Integer>"]
MapResults --> Return["Return Filter Options"]
Each filter method follows a consistent structure:
SELECT field, COUNT(*) as count
FROM "devices"
WHERE <dynamic conditions>
GROUP BY field
ORDER BY count DESC
Supported filter dimensions:
When generating filter options for a specific field (e.g., status), the repository:
This enables faceted search behavior where selecting one filter dynamically updates other filter counts.
SELECT COUNT(*)
FROM "devices"
WHERE <dynamic conditions>
This method ensures:
Package: com.openframe.data.repository.pinot
PinotClientLogRepository is the primary analytical repository for log search and event querying.
It supports:
flowchart TD
Client["API / GraphQL"] --> Repo["PinotClientLogRepository"]
Repo --> Builder["PinotQueryBuilder"]
Builder --> PinotExec["Pinot Broker"]
PinotExec --> ResultSet["ResultSet"]
ResultSet --> Mapper["Row Mapper Functions"]
Mapper --> Projection["LogProjection / OrganizationOption"]
Uses PinotQueryBuilder for:
whereDateRange() whereIn() whereEquals() whereCursor() whereRelevanceLogSearch() orderBySortInput()This ensures:
Cursor filtering ensures stable pagination using:
toolEventId eventTimestampThis guarantees deterministic ordering.
Sortable columns are whitelisted:
- eventTimestamp
- severity
- eventType
- toolType
- organizationId
- deviceId
- ingestDay
Invalid sort fields are rejected.
Rows are mapped dynamically using a column index map.
flowchart LR
ResultSet["ResultSet"] --> IndexMap["Column Index Map"]
IndexMap --> Projection["Populate LogProjection"]
Each row maps to:
Distinct query returning:
Null or empty IDs are excluded.
Both repositories:
PinotQueryException flowchart TD
Execute["Execute Pinot Query"] --> Success{"Success?"}
Success -->|Yes| Return["Return Results"]
Success -->|No| LogErr["Log Error"]
LogErr --> Throw["Throw PinotQueryException"]
GROUP BY and DISTINCT PinotQueryBuilder flowchart TD
Stream["Stream Service"] --> Pinot["Apache Pinot"]
Pinot --> DataPinot["Data Pinot Repositories And Models"]
DataPinot --> ApiCore["API Service Core"]
ApiCore --> Frontend["Frontend / External API"]
The Data Pinot Repositories And Models module is the analytical backbone of OpenFrame’s real-time querying capabilities.
It provides:
By leveraging Apache Pinot and structured query construction patterns, this module enables scalable, low-latency analytics across tenant environments while maintaining clean repository abstractions and strong query safety controls.