# Machbase DBMS Manual — Full Text
> Markdown corpus of the current published English DBMS 8.7.0 pages in navigation order.
- language: `en`
- document_count: `242`
- document_index: https://docs.machbase.com/llms-chunks.json
---
title: "Machbase DBMS Manual"
url: https://docs.machbase.com/dbms/
language: en
kind: section
---
# Machbase DBMS Manual
This manual covers installation, table-type usage, application development, operations, security,
and reference information for Machbase 8.7.0.
If you are new to Machbase, work through the SQL exercises in Chapter 1, then learn data models
and storage and operational principles in Chapter 2. For system design, review table selection
in Chapter 4 before moving to the relevant table chapter. For exact syntax or feature support,
use Chapter 16.
## Manual Structure
| Chapter | Title | Scope |
|----|------|------|
| 1 | [Getting Started](./getting-started/) | Overview, connection checks, quick start, basic commands |
| 2 | [Core Concepts](./core-concepts/) | Table types, time model, ROLLUP, and retention policies |
| 3 | [Installation, Deployment, and Upgrade](./installation-deployment-upgrade/) | Preparation, Standard Edition, Cluster Edition, upgrade |
| 4 | [Table Type Selection and Schema Design](./data-modeling-table-design/) | Type decisions, schema, mutation policy, and patterns |
| 5 | [TAG Table Usage](./tag-table-usage/) | TAG structure, metadata, input, query, correction, operations |
| 6 | [ROLLUP for TAG Tables](./tag-rollup-usage/) | ROLLUP design, creation, query, rebuild, operations, performance tuning |
| 7 | [LOG Table Usage](./log-table-usage/) | LOG structure, input, and text search |
| 8 | [TRANSACTION Table Usage](./rdb-table-usage/) | TRANSACTION schema, DML, transactions, JOIN, backup and recovery |
| 9 | [LOOKUP Table Usage](./lookup-table-usage/) | Reference data, primary keys, JSON, predicate DML, and joins |
| 10 | [VOLATILE Table Usage](./volatile-table-usage/) | Memory tables, UPSERT, state cache, restart and data loss |
| 11 | [Development and Application Integration](./development-tools-integration/) | Integration methods, common concepts, and language-specific SDKs/APIs |
| 12 | [Performance Tuning](./performance-tuning/) | Query, ingestion, and cache tuning |
| 13 | [Operations, Configuration, and Recovery](./operations-configuration-recovery/) | Server operations, backup, and Cluster operations |
| 14 | [Accounts, Privileges, and Access Control](./security-access-control/) | Accounts, privileges, AUTH KEY, and access control |
| 15 | [Troubleshooting](./troubleshooting/) | Diagnostics and resolution guides |
| 16 | [Reference](./reference/) | SQL, functions, configuration, and system catalog reference |
---
title: "1. Getting Started"
url: https://docs.machbase.com/dbms/getting-started/
language: en
kind: section
---
# 1. Getting Started
This chapter is a starting point for readers new to databases and for developers coming from
another DBMS. It introduces how Machbase DBMS 8.7.0 stores data, then walks through the basic SQL
workflow by inserting and querying one event.
You can read the overview before installing a database. To run the example, you need a running
DBMS server and its SQL client, `machsql`. The prerequisites in
[10-Minute Quick Start](./quick-start/) explain how to prepare the server and connection.
## What You Will Learn
1. Understand the roles of tables, rows, columns, and SQL.
2. Distinguish historical records from current state.
3. Identify Machbase table types for measurements, events, and reference data.
4. Connect to the server, create a LOG table, and insert and query data.
5. Compare event time with server arrival time and choose your next learning path.
## Reading Order
| Section | What you can do afterward |
|---|---|
| [Machbase DBMS Overview](./overview/) | Explain the purpose of a time-series database and the roles of its table types. |
| [10-Minute Quick Start](./quick-start/) | Connect, insert and query data, and clean up the practice table. |
| [Basic Command Cheatsheet](./command-cheatsheet/) | Distinguish shell commands from SQL and find common commands. |
| [Choose the Next Document](./choose-next-doc/) | Find the documents relevant to your data, development tasks, and operations. |
The example uses a small SQL `INSERT` so you can inspect the result directly. For continuous,
high-volume ingestion and error handling, continue with
[Data Input and Export](/dbms/development-tools-integration/data-input-load-export/).
---
title: "1.1 Machbase DBMS Overview"
url: https://docs.machbase.com/dbms/getting-started/overview/
language: en
kind: page
---
# 1.1 Machbase DBMS Overview
Machbase DBMS is a time-series database for storing and analyzing data that accumulates over time,
such as sensor measurements, equipment events, and application logs. You choose
tables to match the structure of your data and how you update and query it, and manage historical
records together with the reference data needed to interpret them.
## Introduction to Machbase DBMS
### What a Database and SQL Do
A database stores application data in a defined structure so that different tasks can reuse it.
A database management system, or DBMS, is the software that manages requests to read and write
that data, user permissions, and storage space.
A table groups records with the same structure. A row can represent one event or measurement.
Columns hold individual attributes, such as a timestamp, equipment name, or measured value.
Each column has a data type, such as a number, string, or date and time. This defined structure
is called a schema.
For example, a temperature history might look like this. The times and values below are
illustrative.
| Measurement target | Measurement time | Temperature |
|---|---|---:|
| Equipment A temperature sensor | 09:00:00 | 23.1 |
| Equipment A temperature sensor | 09:00:01 | 23.5 |
| Equipment B temperature sensor | 09:00:01 | 18.0 |
SQL is the language used to work with this data. `CREATE` creates a table, `INSERT` adds rows,
and `SELECT` reads the rows and columns you need. `WHERE` specifies selection conditions,
`GROUP BY` defines aggregation groups, and `ORDER BY` controls result ordering.
Although the SQL language is shared, supported changes and storage behavior depend on the
table type.
### Current State and Historical Records
“What is the current temperature of equipment A?” and “How did its temperature change over the
past hour?” are different questions. Repeatedly overwriting one current value loses the history.
Storing each measurement as a new row with a timestamp lets you analyze trends, maximum values,
and when an abnormal condition occurred.
These histories keep growing in a time-series system. In addition to ingestion speed, you must
decide which targets and time ranges you will query and how long you need to keep raw data.
Measurements do not necessarily arrive at regular intervals. Network delays, equipment outages,
and retransmission can produce late or missing readings.
### Measurements, Events, and Reference Data
A measurement describes the value of a particular target at a particular time. An event records
something that happened, such as an alarm, an equipment stop, or a service starting. Equipment
names, installation locations, and units of measurement are reference data used to interpret
those records.
A system often needs all three. Investigating an abnormal temperature may require the temperature
history, alarms from the same period, and the sensor's installation location. SQL joins
(`JOIN`) connect historical records with reference data through a shared value, such as an
equipment identifier.
## Problems Machbase Solves
Machbase's time-series features can be used together for the following tasks.
| Task | Example | Related features |
|---|---|---|
| Collect raw history | Continuously store sensor readings and equipment events | TAG and LOG tables, SQL input, and SDK Append |
| Query a relevant range | Inspect equipment A's temperature changes over the past hour | Tag and time conditions, indexes, and execution plans |
| Query recurring statistics | Compare minute or hourly trends over long periods | TAG ROLLUP |
| Interpret data | Associate sensor codes with equipment names and locations | Reference data and JOIN |
| Manage retention periods | Remove raw records past their retention period | Retention Policy on supported tables |
| Protect data against failures | Verify backup and recovery procedures | Edition-specific backup and recovery features |
ROLLUP computes statistics over ranges of raw data. Compression reduces the space needed to
store data, while a Retention Policy deletes old records. These features serve different purposes;
having aggregates does not by itself mean that it is appropriate to delete the raw data.
Performance requirements depend on data types, ingestion volume, concurrent queries, retention
periods, and server resources. Learn the workflow with a small example, then measure throughput
and latency using actual data and query conditions.
[Core Concepts](../../core-concepts/) explains the roles of these features in more detail.
## Choosing a Table for Your Data
| Table | Main use | What to check when choosing |
|---|---|---|
| TAG | Measurement histories organized by tag name and a time or distance axis | Whether queries focus on ranges and aggregates for specific tags. |
| LOG | Event and log histories with multiple attributes | Whether you continuously add records and read them using time or search conditions. |
| LOOKUP | Persistent reference data, such as equipment codes and mappings | The size and update pattern of the reference data loaded into memory. |
| TRANSACTION | Business data requiring row-level changes and transactions | Whether you need relational DML and transactions in Standard Edition. |
| VOLATILE | Shared state that can be recreated after a restart | Whether you can rebuild the data when it is lost at server shutdown. |
In Machbase DBMS 8.7.0, explicitly use `CREATE LOG TABLE` to create a LOG table. A bare
`CREATE TABLE` creates a TRANSACTION table, which is supported in Standard Edition.
An industry label alone does not determine the table type. For the final choice, including
update, join, and retention requirements, see
[Table Type Selection](/dbms/data-modeling-table-design/table-types-selection-type/).
Next, store one event and read it back in [10-Minute Quick Start](../quick-start/).
---
title: "1.2 10-Minute Quick Start"
url: https://docs.machbase.com/dbms/getting-started/quick-start/
language: en
kind: page
---
# 1.2 10-Minute Quick Start
Connect to a running server, store one service-start event, and read it back. This example uses
a LOG table for append-oriented event data and explains the SQL results and the two time columns.
The estimated time does not include installing the server.
## Prerequisites
- Machbase DBMS is running on `127.0.0.1:5656`.
- The `machsql` command is available.
- You can connect with a practice account that can create, insert into, query, and drop tables.
- The commands below use the initial practice account `SYS` and password `MANAGER`. If the
password has been changed, substitute its current value.
- You can save a SQL file under `/tmp`, and the practice table name `DBMS_GS_QUICK` is available.
Use a practice environment where this name does not conflict with a business table.
The final `DROP TABLE` deletes the practice table and the data inserted into it.
If the server is not ready yet, start with
[Installation, Deployment, and Upgrade](/dbms/installation-deployment-upgrade/) and
[Linux Standard Edition Installation](/dbms/installation-deployment-upgrade/standard-edition/#linux).
## Representative Sample
This sample records one service-start event in a LOG table. Explicitly create it with
`CREATE LOG TABLE`; the `_arrival_time` column is added automatically.
Save the SQL file and run it with the following commands.
```bash
cat > /tmp/dbms_gs_quick.sql <<'SQL'
CREATE LOG TABLE DBMS_GS_QUICK (
EVENT_ID INTEGER,
EVENT_TIME DATETIME,
LEVEL VARCHAR(10),
MESSAGE VARCHAR(40)
);
INSERT INTO DBMS_GS_QUICK (EVENT_ID, EVENT_TIME, LEVEL, MESSAGE)
VALUES (
1,
TO_DATE('2026-07-02 09:00:00', 'YYYY-MM-DD HH24:MI:SS'),
'INFO',
'service started'
);
SELECT _arrival_time, EVENT_ID, EVENT_TIME, LEVEL, MESSAGE
FROM DBMS_GS_QUICK
ORDER BY EVENT_ID;
DROP TABLE DBMS_GS_QUICK;
SQL
machsql -s 127.0.0.1 -P 5656 -u SYS -p MANAGER -f /tmp/dbms_gs_quick.sql
```
### Check the Results
Check each SQL step for errors. The SELECT result should contain one row with `EVENT_ID`
equal to `1`, `LEVEL` equal to `INFO`, and `MESSAGE` equal to `service started`.
`EVENT_TIME` is the event timestamp supplied by the application. In this example,
`_arrival_time` is the server arrival timestamp recorded automatically by the DBMS.
Its value therefore changes each time you run the example and need not equal `EVENT_TIME`.
`CREATE LOG TABLE` defines the structure, and `INSERT` adds one row. `SELECT` specifies the
columns to read, while `ORDER BY EVENT_ID` specifies result ordering. If the final
`DROP TABLE` succeeds, the practice table is removed. To inspect the data further, omit the
final DROP statement before running the script, then remove only that practice table when
you finish.
If a rerun fails because `DBMS_GS_QUICK` already exists, a previous run may not have reached
the `DROP TABLE` step. Inspect it with `DESC DBMS_GS_QUICK;`. Only if it is the table from
your previous practice run should you execute `DROP TABLE DBMS_GS_QUICK;` and retry.
For a connection error, first check the server address, port, running state, and account details.
## What This Sample Checks
| Item | What it verifies |
|---|---|
| Server connection | `machsql` connects to `127.0.0.1:5656`. |
| Table creation | `CREATE LOG TABLE` explicitly creates a LOG table. |
| Data input | `INSERT` and `TO_DATE` store event data. |
| Data query | `SELECT` and `ORDER BY` verify the inserted row. |
| Automatic column | The server automatically records the LOG table's `_arrival_time`. |
| Cleanup | `DROP TABLE` removes the practice table. |
---
title: "1.3 Basic Command Cheatsheet"
url: https://docs.machbase.com/dbms/getting-started/command-cheatsheet/
language: en
kind: page
---
# 1.3 Basic Command Cheatsheet
Run connection commands in your operating system's terminal and SQL in a connected `machsql`
session. Adjust the server address, port, and account for your environment. `MANAGER` in these
examples is the initial practice password used in the quick start. If it has been changed, use
the account's current password.
## Commands to Run in the Terminal
| Task | Command |
|---|---|
| Connect interactively | `machsql -s 127.0.0.1 -P 5656 -u SYS -p MANAGER` |
| Run a SQL file | `machsql -s 127.0.0.1 -P 5656 -u SYS -p MANAGER -f file.sql` |
## SQL to Run in machsql
| Task | Command |
|---|---|
| List tables | `SHOW TABLES;` |
| Inspect columns and types | `DESC table_name;` |
| Create a TRANSACTION table | `CREATE TRANSACTION TABLE table_name (...);` |
| Create a LOG table | `CREATE LOG TABLE table_name (...);` |
| Insert data | `INSERT INTO table_name VALUES (...);` |
| Query matching data | `SELECT ... FROM table_name WHERE ...;` |
| Sort query results | `SELECT ... FROM table_name ORDER BY ...;` |
| Delete a table and its data | `DROP TABLE table_name;` |
`table_name` and `...` are placeholders that must be replaced with actual names and definitions.
For runnable SQL, use [Quick Start](../quick-start/).
`DROP TABLE` also deletes the table's data, so first confirm that it is the practice table you
intend to remove.
In Machbase DBMS 8.7.0, `CREATE TABLE` without a table type creates a TRANSACTION table.
TRANSACTION tables are supported in Standard Edition. Specifying the type makes the intent of
an example clear. For the complete command reference, see
[machsql](../../reference/command-line-tools/machsql/).
---
title: "1.4 Choose the Next Document"
url: https://docs.machbase.com/dbms/getting-started/choose-next-doc/
language: en
kind: page
---
# 1.4 Choose the Next Document
The quick start stored and queried one row. When building a system, you also need to decide
how data changes, which timestamp matters, how it will be queried, and how long to retain it.
## Questions to Answer Before Designing
- Do you accumulate values over time for the same target, or overwrite its current state?
- Do queries use measurement or event time, or the time the server received the data?
- Which operations are most common: ranges for a specific tag, searches across columns, key
lookups, or joins?
- How will you handle late arrivals, duplicates, and incorrect values?
- How long do you need raw data and aggregates, and which data must survive a restart?
- Which Edition supports the features you need?
## Next Paths
| What you need to do | Next document |
|---|---|
| Understand history, storage, and querying | [Core Concepts](../../core-concepts/) |
| Choose table types and update policies | [Table Type Selection and Schema Design](../../data-modeling-table-design/) |
| Store sensor and measurement histories | [TAG Tables](../../tag-table-usage/) |
| Collect and search events and logs | [LOG Tables](../../log-table-usage/) |
| Query tag statistics over long periods | [ROLLUP for TAG Tables](../../tag-rollup-usage/) |
| Manage reference and business data | [LOOKUP](../../lookup-table-usage/), [TRANSACTION](../../rdb-table-usage/) |
| Manage state that can be recreated | [VOLATILE Tables](../../volatile-table-usage/) |
| Insert and query data from applications | [Development and Application Integration](../../development-tools-integration/) |
| Check SQL syntax and data types | [SQL Reference](../../reference/sql/) |
| Configure operations, backups, and permissions | [Operations, Configuration, and Recovery](../../operations-configuration-recovery/), [Accounts and Permissions](../../security-access-control/) |
Start on a small scale with representative data and common queries. Measure ingestion speed,
query latency, and storage use before adjusting ingestion methods, indexes, aggregates, and
retention policies.
---
title: "2. Core Concepts"
url: https://docs.machbase.com/dbms/core-concepts/
language: en
kind: section
---
# 2. Core Concepts
Chapter 1 introduced the SQL workflow for storing and reading data. This chapter explains that
workflow from a design and operations perspective. The appropriate table and storage strategy
depend on the meaning of time, how data changes, and which ranges you query.
For example, keeping a device's current temperature requires a different number of records
from keeping its temperature history for the past month. Calculating a monthly average and
detecting brief anomalies require different levels of detail. Understanding these differences
helps you choose tables, indexes, ROLLUP, and retention policies for their intended purposes.
## Contents
| Section | Questions it addresses |
|---|---|
| [Data Model Concepts](concepts/) | What does one row represent, and how should time, NULL, duplicates, and changes be interpreted? |
| [Storage and Execution Architecture](storage-execution-architecture/) | How is relevant data found, and which costs do storage, indexes, and caches reduce? |
| [Feature Concepts and Distinctions](features-concepts/) | What are the separate purposes of raw data, aggregates, retention, and backups? |
| [Edition Concepts](concepts-edition/) | When should you choose a single server or a distributed deployment, and which feature differences matter? |
The examples use small data sets to explain concepts. Follow the links in each section for
feature-specific SQL, support limits, and operational procedures. Then apply the concepts to your
own data in [Table Type Selection and Schema Design](../data-modeling-table-design/).
---
title: "2.1 Data Model Concepts"
url: https://docs.machbase.com/dbms/core-concepts/concepts/
language: en
kind: page
---
# 2.1 Data Model Concepts
A data model defines what one record represents, how records are identified, and how they are
changed and queried. A time-series model also defines the meaning of the measurement target,
timestamp, and value. Establishing these rules before choosing column names helps keep ingestion
and analysis consistent.
## Understanding Time-Series Data
Time-series data records observations or events over time. Examples include temperature
measurements, trade histories, and service errors. Records may be collected at fixed intervals or
only when events occur. Storage order is not guaranteed to match event order.
### What One Row Represents and How It Is Identified
If one temperature-history row means “the value measured by one sensor at a particular time,”
you need a sensor identifier, measurement timestamp, and value. The identifier tells you where
the data came from, while the timestamp provides a basis for interpreting changes over time.
Multiple sensors can report at the same time, so a timestamp alone cannot uniquely identify a row.
Retransmission can also repeat a reading for the same sensor and timestamp. Decide whether to
allow duplicates, remove them, or introduce a separate event identifier. A TAG `PRIMARY KEY`
identifies a tag; it does not require every measurement row to have a unique name as a relational
row key would. See [TAG Table Operations](../../tag-table-usage/operations-lifecycle/) for
duplicate handling.
### Values, Units, and Quality
A number alone does not explain its unit or meaning. The same `23.5` could represent temperature,
pressure, or voltage. Manage units and measurement locations in tag definitions or reference
data, and keep the meaning of values consistent within a series.
Also distinguish `NULL` from `0`. Measuring zero is different from failing to obtain a value.
Record quality or status in another column when needed. A period with no collection may have no
rows at all, which is different from rows containing NULL.
Common aggregates such as `AVG` operate on non-NULL values. Filling all missing values with zero
or aggregating values with different units changes the result. For the exact NULL behavior of
supported aggregates, see the [Function Reference](../../reference/sql/functions/functions-full/).
### Time-Series Workload Characteristics
Many time-series systems continuously add rows and query time ranges for particular targets.
They may combine monitoring of current values with analysis over long periods. These patterns
motivate append-oriented ingestion, range queries, and ROLLUP.
Historical data is not necessarily immutable, however. Incorrect equipment clocks, sensor
calibration, or duplicate collection may require corrections. Some workloads read older data
more often than recent data. Design around measured ingestion, query, and correction patterns.
### Table-Type Roles
| Table type | Conceptual role |
|---|---|
| TAG | Measurement histories organized by name and a time or distance axis |
| LOG | Events and logs that continuously accumulate new records |
| TRANSACTION | Business data requiring transactions and row-level changes |
| LOOKUP | Persistent reference data loaded into memory |
| VOLATILE | Shared in-memory state that can be recreated after a restart |
Separating history from reference data reduces repetition of equipment names or locations in
every measurement row. However, joining historical records to current reference data also shows
current names or locations. If you need information as it was when an event occurred, store it
with the event or design a separate history of reference-data changes.
### Relational Business Models and Time-Series Models
Relational and time-series models are not mutually exclusive. A relational DBMS can store time
series and use indexes, partitions, and aggregates. Machbase also provides tables, SQL, joins, and
relational changes. Compare the main workload rather than the product names.
| Perspective | Relational business-data example | Time-series history example |
|---|---|---|
| Meaning of a row | The current state of an order | One sensor measurement or event |
| Change pattern | Update or delete a row found by key | Add records and correct or remove relevant ranges |
| Query pattern | Key lookups, condition searches, and joins of business tables | Target and time-range queries, trends, and interval statistics |
| Consistency requirements | Commit or cancel several changes together | Manage missing, duplicate, or delayed input and when it becomes queryable |
| Retention design | Business lifecycle and change history | Raw-data resolution, aggregation intervals, and retention periods |
Do not apply the storage and ingestion characteristics of LOG and TAG unchanged to TRANSACTION,
LOOKUP, or VOLATILE. For the final choice, see
[Table Type Selection](../../data-modeling-table-design/table-types-selection-type/) and
[Data Mutation Policy](../../data-modeling-table-design/alter-data-mutation-policy/).
## Write-Oriented Workloads and the Append-Only Model
Appending adds a new row instead of overwriting an existing value. For example, recording a new
event when equipment changes from `RUNNING` to `STOPPED` preserves both the previous state and
the transition time. You can also maintain separate tables for direct access to current state
and for its history of changes.
### Table Types and Change Models
LOG is an append-oriented history table and does not support general row `UPDATE`. TAG adds
measurement history and can correct DATA values within supported conditions and Edition limits.
This differs from a general row update that can arbitrarily change tag names or the time axis.
TRANSACTION provides relational DML and explicit transactions. LOOKUP and VOLATILE support
reference-data or state changes. Check
[Data Mutation Policy](../../data-modeling-table-design/alter-data-mutation-policy/) rather than
assuming that every table type supports the same operations.
Append-oriented design can help reduce contention between repeated input and arbitrary updates
to historical rows. It does not eliminate all internal synchronization or recovery work.
A storage structure alone does not guarantee lock-free execution or a particular throughput.
### SQL Input and SDK Append
SQL `INSERT` submits values through a SQL statement and reports its execution result.
SDK Append is an ingestion API for sending multiple rows during continuous collection.
Follow the SDK's contract for buffering, transmission, error checks, and closing resources.
Putting data into an application buffer, having the server process it, and reaching a point from
which it can be recovered after a failure are different events. Append operations on LOG and TAG
are not rolled back by `ROLLBACK` in an explicit TRANSACTION-table transaction. For Append into
TRANSACTION tables, check your SDK's transaction participation and error-handling contract.
See [Data Input and Export](../../development-tools-integration/data-input-load-export/) for
completion checks and retries.
### Corrections, Schema Changes, and Retention
One way to preserve a correction to an incorrect LOG event is to append a correction event.
Before deleting and reinserting data, check the time-based deletion ranges and input ordering
allowed for LOG. Do not assume that any individual row can be deleted independently.
For TAG values, follow
[TAG Data Correction Design](../../tag-table-usage/tag-data-update-correction/#design-correction-tag).
Adding or dropping columns and changing types are separate capabilities. Schema changes are not
universally prohibited for LOG or TAG; support depends on the table type and DATA or METADATA
area. Check [DDL Syntax](../../reference/sql/syntax/ddl-syntax/) before applying
changes.
Continuously appending raw records increases storage use. Define raw-data retention separately
from the purpose of ROLLUP statistics. Creating aggregates does not by itself delete raw data.
## Time Models and _arrival_time
### Event Time and Arrival Time
Event time is when a device made a measurement or an event occurred. Arrival time is when the
DBMS received the record. If a reading measured at 09:00 arrives at 09:05 after network recovery,
the difference is five minutes. Measurement trends require event time; collection-delay analysis
requires comparing the two timestamps.
Also distinguish precision, time zone, and clock accuracy. Being able to represent nanoseconds
does not mean a sensor clock is accurate to a nanosecond. For the time zone used when parsing or
displaying DATETIME strings, see
[Time Zone Configuration](../../reference/configuration/configuration-timezone/).
### The LOG Table's `_arrival_time`
LOG automatically includes a DATETIME column named `_arrival_time`. With ordinary input that
omits its value, the server records the arrival timestamp. Define a separate DATETIME column
if you also need event time.
```sql
CREATE LOG TABLE device_events (
device_id VARCHAR(20),
event_time DATETIME,
status VARCHAR(20)
);
```
Some input paths explicitly supply `_arrival_time`, so it cannot always be treated as the
actual time of receipt. For ordering and restrictions on explicit values, see
[The LOG Time Model](../../log-table-usage/arrival-time-model/).
```sql
SELECT device_id, event_time, status
FROM device_events
WHERE _arrival_time >= TO_DATE('2026-07-03 09:00:00', 'YYYY-MM-DD HH24:MI:SS')
AND _arrival_time < TO_DATE('2026-07-03 10:00:00', 'YYYY-MM-DD HH24:MI:SS')
ORDER BY _arrival_time;
```
A half-open interval `[start, end)` includes the start and excludes the end, helping avoid
counting shared boundaries twice when combining adjacent intervals. `BETWEEN` includes both
boundaries; choose the form that matches your purpose. LOG `DURATION` restricts a range using
`_arrival_time`.
### The TAG Table's BASETIME Column
In a time-axis TAG table, the application supplies timestamps in a DATETIME column declared
with `BASETIME`. TAG does not use LOG's automatic `_arrival_time` column.
```sql
CREATE TAG TABLE sensor_values (
name VARCHAR(128) PRIMARY KEY,
time DATETIME BASETIME,
value DOUBLE SUMMARIZED
);
INSERT INTO sensor_values
VALUES ('temp_sensor_01',
TO_DATE('2026-07-03 08:55:00', 'YYYY-MM-DD HH24:MI:SS'),
23.1);
```
Here, `name` identifies the sensor, `time` is the measurement timestamp, and `value` is the
measured value. `SUMMARIZED` designates a representative numeric column for related statistics
features such as ROLLUP. Creating the table alone does not create aggregates for every interval
you might need.
TAG also has a BASE DISTANCE model with a numeric axis instead of time. Consider it for
measurements along a distance or position axis, but do not apply time-axis-only functions and
policies unchanged. [TAG Schema](../../tag-table-usage/table-structure-schema/) explains the rules
for each axis.
### Comparing the Two Time Models
| Item | LOG | Time-axis TAG |
|---|---|---|
| Special time column | Automatically created `_arrival_time` | Declared `BASETIME` column |
| Meaning of the input timestamp | Server time by default, or an explicitly supplied value | Time specified by the application |
| Separate event time | Can be stored in an ordinary DATETIME column | BASETIME can represent event time |
| Main design question | Which events and fields will be searched? | Which range of which tag will be analyzed? |
Needing event time does not by itself rule out LOG. Choose a table based on tag structure,
searches, aggregates, and update and deletion conditions, as well as the meaning of time.
DATETIME columns in LOOKUP, VOLATILE, and TRANSACTION are ordinary columns; they do not
automatically acquire the special time-axis behavior of LOG or TAG.
Distinguish storage order from display order, too. Specify `ORDER BY` when result ordering
matters, and add a tie-breaker when rows with identical timestamps must be ordered.
---
title: "2.2 Storage and Execution Architecture"
url: https://docs.machbase.com/dbms/core-concepts/storage-execution-architecture/
language: en
kind: page
---
# 2.2 Storage and Execution Architecture
Query time is not determined by the length of the SQL statement alone. It also depends on how
much data is read, how effectively conditions narrow that data, and the work required for sorting,
joins, and aggregation. This section explains which costs storage, indexes, and caches reduce.
## Machbase Architecture Overview
Users send write and query requests through `machsql` or an SDK. The server checks SQL syntax,
objects, and permissions, chooses an execution method, accesses stored data, and returns the
results. Storage and execution behavior depend on the table type and Edition.
### Standard Edition Architecture
Standard Edition handles SQL processing and storage in a single database server. Time-series
ingestion into LOG and TAG, relational changes in TRANSACTION, and reference data or state in
LOOKUP and VOLATILE each use a path suited to the table's characteristics.
```text
Client: machsql or SDK
| Write and query requests
v
Machbase DBMS server
SQL analysis and execution plans
Table-specific storage and index access
Memory buffers and background processing
|
v
Stored data organized by table type
```
This is a conceptual diagram of the roles involved. It does not mean that every request uses
the same storage path or that an API call immediately writes a disk file.
### Cluster Edition Architecture
Cluster Edition divides work among several node roles. Ordinary application SQL connections
use a Broker, while Warehouses store time-series data and execute queries. Coordinators manage
cluster metadata and node state, Lookup nodes process reference data, and Deployers handle
deployment and node management.
Distributing data across groups shares processing and storage load. Replication within a group
prepares for failures. These are different goals: adding nodes does not make every query faster
by the same factor or automatically recover from every failure.
For feature differences, see [Edition Concepts](../concepts-edition/). Deployment and recovery
procedures are covered in [Cluster Installation](../../installation-deployment-upgrade/cluster-edition/)
and [Cluster Operations](../../operations-configuration-recovery/cluster/).
### Write and Query Flows
A write can be understood as checking the target table and columns, converting and validating
values, and transferring and storing data. SQL and Append APIs report processing results
differently, so applications must follow the error-handling and completion rules of their chosen
path.
A query involves SQL analysis and planning, data access, condition evaluation, aggregation, joins,
sorting, and result delivery. Not every query processes data in exactly that order; the actual
execution plan determines the access order.
## Columnar Storage and Compression
### Row-Oriented and Column-Oriented Storage
Row-oriented storage groups values belonging to one row. Column-oriented storage groups values
from the same column. The following illustration compares the ideas; it is not a literal diagram
of the on-disk files.
```text
Logical rows:
(time1, sensorA, 23.1)
(time2, sensorA, 23.5)
(time3, sensorB, 18.0)
Row-oriented: [time1, sensorA, 23.1] [time2, sensorA, 23.5] ...
Column-oriented: [time1, time2, time3] [sensorA, sensorA, sensorB] [23.1, 23.5, 18.0]
```
Machbase's LOG and TAG time-series storage uses column-level access and compression. This can
reduce the amount of data read when an analysis needs only a few columns across many rows.
However, even a temperature average may also need sensor and timestamp data to evaluate its
conditions.
Row-oriented systems can also use indexes or partitions to read only relevant ranges. Compare
the number of rows and columns read and the access path, rather than assuming that row-oriented
storage always reads every row or that column-oriented storage is always faster. The relational
storage of TRANSACTION and the memory characteristics of LOOKUP and VOLATILE should not be
interpreted as identical to LOG and TAG storage.
### When Compression Helps
Values in a column share a type, and sensor values or timestamps may exhibit repeated or similar
patterns. These characteristics can help compression. Noisy values, irregular strings, and mixed
input ordering can produce different results.
Time-series timestamps are not necessarily monotonically increasing. Late measurements and input
from multiple sources may be interleaved. Measure compression and processing performance with the
actual types, value distributions, input order, and settings, rather than assuming a particular
algorithm or compression ratio.
### Partitions and Read Ranges
A partition divides data into manageable portions. Using query conditions and stored range
information to skip unrelated portions reduces reads. This is called partition pruning.
LOG and TAG storage units do not necessarily correspond to a user-defined day or month.
The actual access range depends on tag and axis conditions, data distribution, and each table's
storage structure. A time condition alone does not establish that only the necessary data is read;
check execution plans and measurements.
## Indexing Principles
An index is an access path for finding data that matches a condition. It can help when the target
is a small fraction of the data, but another path may be better for an aggregate that reads most
rows. Indexes also require storage and maintenance during writes and updates.
| Table type | Basis for considering access paths |
|---|---|
| TAG | Tag name and ranges on the time or distance axis |
| LOG | `_arrival_time` conditions and supported search indexes |
| TRANSACTION | PRIMARY KEY, UNIQUE, and general indexes |
| LOOKUP and VOLATILE | Memory-resident keys and supported secondary indexes |
“The monthly average for all sensors” and “the last minute of values for sensor A” read different
proportions of the data. Even on the same table, their performance can differ substantially.
For joins, the input row counts and join conditions also matter.
See [Schema Object Definitions](../../data-modeling-table-design/schema-objects-definition/) for
supported indexes and restrictions, and [Index Tuning](../../performance-tuning/index-tuning/)
for measurement and adjustment.
## Caches and Execution Plans
### SQL Execution
The server checks SQL syntax, types, and objects, identifies executable access paths using
conditions and indexes, and processes the actual data. An execution plan describes that method.
The cost of building a plan differs from the cost of reading data according to the plan.
### Plan Reuse and PVO Cache
The PVO Statement Cache reuses parsing, validation, optimization results, and execution plans
for reusable SQL, reducing repeated preparation work. It does not cache result rows: reusing a
plan still requires reading data and evaluating conditions.
Caches such as Min-Max Cache use stored range information to reduce the data that needs to be
read. Distinguish execution-plan caching from caches used for data access. Decide which cache
to enlarge after examining hit rates and memory use.
### Inspecting a Plan with EXPLAIN
The following example uses the `sensor_values` table created in
[Data Model Concepts](../concepts/#time-model-arrival-time).
```sql
EXPLAIN SELECT AVG(value)
FROM sensor_values
WHERE name = 'temp_sensor_01'
AND time >= TO_DATE('2026-07-01', 'YYYY-MM-DD')
AND time < TO_DATE('2026-07-03', 'YYYY-MM-DD');
```
Use the plan to inspect data access and condition evaluation. A plan alone does not show the
actual response time or amount of disk I/O, so also measure execution with representative data.
Distinguishing a first run with empty caches from a run that reuses cached information makes the
results easier to interpret.
Check PVO cache hit and eviction statistics in `V$PVO_CACHE_STAT` and cached SQL in
`V$PVO_CACHE_LIST`. For configuration and diagnosis, see
[Cache and Memory Tuning](../../performance-tuning/cache-tuning-memory/#pvo-cache).
---
title: "2.3 Feature Concepts and Distinctions"
url: https://docs.machbase.com/dbms/core-concepts/features-concepts/
language: en
kind: page
---
# 2.3 Feature Concepts and Distinctions
This section explains the roles and selection criteria of ROLLUP, Retention Policy, Backup,
Restore, and Mount for long-term data management in Machbase DBMS. Their detailed documents
provide creation syntax and operational procedures.
Raw data lets you reexamine individual events or measurements. Aggregates summarize many raw
records for recurring queries. Retention policies determine what to delete and when, while
backups provide material for recovery after a failure. Distinguishing these purposes helps
avoid deleting necessary raw data merely because aggregates exist, or omitting backups because
replication is configured.
- **[The Role of ROLLUP Statistics](#role-statistics-rollup)** — Reduce the cost of recurring
aggregate queries.
- **[The Role of Retention Policy](#role-retention-policy)** — Automatically remove data based
on its age.
- **[Backup, Restore, and Mount](#concepts-backup-restore-mount)** — Distinguish data protection,
recovery, and inspection.
## The Role of ROLLUP Statistics
Repeatedly aggregating long TAG histories from raw rows becomes more expensive as the query
range grows. ROLLUP aggregates specified numeric columns and other supported values from
time-axis TAG data into intervals for recurring queries. Its definition determines the columns
and aggregation methods.
Default ROLLUP creates second (SEC), minute (MIN), and hour (HOUR) levels. Specify `WITH ROLLUP`
when creating a table, or use ordinary `CREATE ROLLUP` to select intervals and conditions.
Manage these objects through the public ROLLUP SQL rather than depending on generated object
names or internal storage structures.
### Selection Criteria
| Query pattern | Starting point |
|---|---|
| Repeated minute or hourly statistics over long periods | Consider default ROLLUP |
| Explicit aggregation intervals or filter conditions | Consider ordinary ROLLUP with the required interval and conditions |
| Store the results of your own aggregate SELECT in a separate TAG table | Consider Custom ROLLUP in Standard Edition |
| First or last values are needed | Consider extended ROLLUP |
| Most queries read raw values, or aggregates are rarely queried | Start without ROLLUP and measure execution time |
ROLLUP is not a retention policy that replaces raw data. Design raw-data retention separately
using Retention Policy, and check whether the relevant ROLLUP range must be rebuilt after
correcting TAG data.
### Interpreting Aggregates
Aggregation reduces detail. Keeping only a one-minute average cannot recover a brief anomaly
or the order of individual measurements within that minute. Keeping minima and maxima also
preserves the range, but still does not preserve every detail of the raw data.
Averaging averages from several intervals can differ from the overall average. For example,
if two observations average 10 and eight observations average 20, the overall average is
`(2 × 10 + 8 × 20) / 10 = 18`, not the unweighted average of 15. Use the necessary statistics,
such as sums and valid counts, when aggregating again, and follow the supported ROLLUP query
functions.
ROLLUP processing takes place separately from raw-data ingestion, so the newest raw rows and
their aggregates may become available at different times. For late arrivals or corrected
values, check the raw range, aggregation progress, and whether rebuilding is necessary.
See [TAG ROLLUP](/dbms/tag-rollup-usage/) for detailed creation syntax, query functions, and
rebuilding procedures.
## The Role of Retention Policy
Retention Policy periodically removes LOG or TAG data past its retention period. It manages
storage for continuously growing tables without requiring operators to repeat deletion
commands manually.
| Requirement | Method |
|---|---|
| Continuously remove data older than a defined period | Retention Policy |
| Immediately remove a specific range of incorrect input | `DELETE`, within the table type's supported conditions |
| Empty all data from a supported table | `TRUNCATE TABLE` |
Attaching a retention policy does not mean all old data disappears immediately. Check the
execution interval, supported table types, and actual deletion state. Manage the lifecycle
of LOOKUP, VOLATILE, and TRANSACTION data using the explicit DML supported by those tables.
Retention and ingestion volume together determine storage needs. Multiplying rows per second
by the retention period in seconds estimates the raw row count, but actual disk use also
depends on data types, compression, indexes, replication, and backups. Before deleting raw
data, check aggregate coverage and retention, and the detail needed for auditing or reanalysis.
Creating ROLLUP does not automatically change raw-data retention.
See [Data Retention Policy](/dbms/operations-configuration-recovery/policy-data-retention/) for
creation, attachment, detachment, and operational checks.
## Backup, Restore, and Mount
All three work with backup data, but their outcomes differ.
| Feature | Purpose | Production server | Result |
|---|---|---|---|
| Backup | Create a copy for recovery | Can run while the server is running | A backup in a separate location |
| Instance Restore | Recover an instance from a backup | Requires an offline procedure | The production database is recovered |
| Mount | Inspect a backup read-only | Can run while the server is running | The backup is queryable under a separate name |
Separately, `RESTORE DATABASE` SQL restores a logical database on a running server.
Distinguish its target and procedure from offline instance restoration.
A successful Backup does not by itself validate the recovery procedure. On a supporting
Edition, inspect the contents with Mount or test Restore in an isolated environment. Also
manage backup-path permissions and retention. Mount does not restore data into the production
database, and mounted data cannot be written. Restore and Mount are Standard Edition features;
in Cluster, follow that Edition's backup and failure-recovery procedures.
Define a recovery point objective (RPO), the acceptable window of data loss, and a recovery
time objective (RTO), the time allowed to restore service. RPO guides review of backup and
replication intervals; RTO guides review of the data to recover and the measured restoration
time. An Edition or backup schedule alone does not guarantee either objective. An incorrect
deletion can also reach replicas, so distinguish replication from backups.
See [Backup, Restore, and Mount](/dbms/operations-configuration-recovery/backup-restore-mount/)
for commands, permissions, Edition support, and recovery sequences. If you operate several
logical databases, also see
[Multi-Database Operations](/dbms/operations-configuration-recovery/multi-database/).
## Comparing Input Paths
Consider `INSERT` for small SQL exercises, a supported SDK's Append API for continuous
application ingestion, and tools such as `machloader` for file loading. Supported tables,
input formats, and failure checks differ by tool, so similar names do not make them
interchangeable.
For selection criteria covering SQL, SDKs, and file-loading tools, see
[Data Input and Export](/dbms/development-tools-integration/data-input-load-export/#machloader-vs-csvimport-csvexport-tagmetaimport).
---
title: "2.4 Edition Concepts"
url: https://docs.machbase.com/dbms/core-concepts/concepts-edition/
language: en
kind: page
---
# 2.4 Edition Concepts
The Edition determines the deployment architecture and supported features. Standard starts with
one server; Cluster divides storage and processing among several nodes. Choose based on the
required SQL features, growth rate, failure recovery, and operational capacity, as well as the
current data size.
## Differences Between Standard and Cluster Editions
### Standard Edition
A single DBMS server handles SQL processing and data storage. You do not need to manage deployment
and communication across distributed nodes, making it suitable for development and single-server
operations. Check whether you need Standard-only features such as TRANSACTION tables, Restore,
or Mount.
Starting with one server does not necessarily mean the data set must be small. Measure whether
the ingestion volume, query workload, and retention period fit within that server's CPU, memory,
and storage capacity.
### Cluster Edition
Coordinator, Deployer, Broker, Warehouse, and Lookup nodes have different roles.
| Node type | Role |
|---|---|
| Coordinator | Manage cluster metadata and node state |
| Deployer | Deploy packages and manage nodes |
| Broker | Accept application SQL connections and distribute queries |
| Warehouse | Store time-series data and execute queries |
| Lookup | Process cluster reference data |
Ordinary SQL applications connect to a Broker. Management tools use role-specific endpoints and
ports, so distinguish SQL connection addresses from management addresses.
Distributing data across Warehouse groups and replicating data within a group serve different
purposes. Distribution expands capacity and throughput; replication helps with failure recovery.
Design node counts, replication state, client reconnection, and recovery procedures together.
### Feature Support Differences
| Item to check | Standard Edition | Cluster Edition |
|---|---|---|
| Main deployment structure | One DBMS server | Multiple nodes with separate roles |
| TRANSACTION tables | Supported | Not supported |
| Restore and Mount | Supported | Not supported |
| Capacity and processing expansion | Expand server resources and storage configuration | Expand distributed groups and node configuration |
| Failure recovery design | Backups, recovery procedures, and server operations | Node and group replication, state, connection handling, and recovery procedures |
Shared features such as LOG, TAG, LOOKUP, VOLATILE, ROLLUP, and Retention do not necessarily have
identical DML, DDL, or operational restrictions. For complete support information, see
[Edition Support](../../reference/support-scope-constraints/edition/) and
[Table Type Support](../../reference/support-scope-constraints/table-types-type/).
### Selection Criteria
1. Identify required features. If you need TRANSACTION tables, Restore, or Mount, first review
Standard Edition support.
2. Run representative ingestion and queries. Use realistic data types, concurrent connections,
query ranges, and retention periods to measure available capacity on one server.
3. Account for data growth and failure requirements. If you need distribution beyond one server,
also consider Cluster's network, replication, and node-management costs.
4. Test failure scenarios. Monitoring nodes does not by itself guarantee uninterrupted recovery.
Measure recovery time and verify application reconnection and retry behavior.
Changing Editions in production involves more than increasing the server count. First verify
compatibility for your SQL and SDKs, data migration, backups, and recovery methods.
## Related Documentation
- [Storage and Execution Architecture](../storage-execution-architecture/) explains the processing
flow.
- [Installation, Deployment, and Upgrade](../../installation-deployment-upgrade/) describes
deployment procedures.
- [Versions and Compatibility](../../reference/support-scope-constraints/compatibility-version/)
lists version-specific support.
- [Relational Business Models and Time-Series Models](../concepts/#differences-rdbms) explains
data-model selection criteria.
---
title: "3. Installation, Deployment, and Upgrade"
url: https://docs.machbase.com/dbms/installation-deployment-upgrade/
language: en
kind: section
---
# 3. Installation, Deployment, and Upgrade
This chapter prepares Machbase DBMS 8.7.0 for use and checks that it can accept and query data.
Installing a new server and upgrading a server that already contains data have different starting
points. First decide which features and deployment environment you need, then choose the matching
procedure.
The data models and Edition differences introduced in Chapter 2 also affect deployment. Check
Standard Edition support if you need TRANSACTION tables, Restore, or Mount. If you need distributed
storage and replication, plan Cluster node roles, networking, and recovery together.
## Choose an Installation Path
| Edition | Deployment structure | What to check |
|---|---|---|
| Standard Edition | One DBMS server handles SQL processing and storage | Required features and whether server resources can support ingestion, queries, and retention |
| Cluster Edition | Separate Coordinator, Deployer, Lookup, Broker, and Warehouse roles | Distribution groups, replication, communication paths, and node operations |
A single server is not limited to small data sets. Measure the required storage and performance
using representative data. Similarly, adding Cluster nodes does not make every query proportionally
faster. See [Edition Differences](/dbms/core-concepts/concepts-edition/#differences-standard-edition-cluster).
### Distinguish the Installation Components
| Component | Meaning | Examples to check |
|---|---|---|
| Distribution package | Executables, libraries, and sample configuration | Version, Edition, OS, and CPU architecture |
| Installation home | Executable and configuration location for a server or node | `MACHBASE_HOME`, `conf/machbase.conf` |
| Data storage path | Location where the DBMS reads and writes data | `DBS_PATH`, free space, and access permissions |
| Server instance or node | DBMS process running with that configuration | Process status, logs, and connection ports |
| Logical database | SQL object namespace selected by a connection | Current database, users, privileges, and tables |
Extracting a package, creating a new instance's database, starting the server, and creating tables
are separate steps. Do not run new-instance initialization against a home containing existing data.
The installation home and actual data path can also differ; check both before backup or upgrade.
A logical database selected through SQL is distinct from an installation home. See
[Multi-Database Operations](/dbms/operations-configuration-recovery/multi-database/) for selecting,
creating, and granting access to multiple databases.
## Installation Sequence
### Standard Edition
1. Use [Pre-Installation Preparation](./pre-install-preparation/) to check the package, server
account, resources, and ports.
2. Prepare a dedicated home and environment following the procedure for your operating system.
- [Linux — Tarball](./standard-edition/#linux-tarball)
- [Linux — Docker](./standard-edition/#linux-docker)
- [Windows — Package](./standard-edition/#windows-package)
3. Check configuration and data paths. If using a separate license, follow
[License Installation](./pre-install-preparation/#license) before the first startup.
4. Create the new database and start the server as described in the selected installation procedure.
5. Use the [Validation Checklist](./validation-checklist/) to check the process, port, connection,
version, license, and a small insert/query workflow.
A container deployment still needs the correct package version, a data volume, and write access
for the server account. The relationship between container startup and DBMS initialization depends
on the image, so follow its installation procedure.
### Cluster Edition
1. Review [Pre-Installation Preparation](./pre-install-preparation/) and
[Cluster Environment Preparation](./cluster-edition/#preparation-environment-cluster-edition).
2. Decide each node's host, home, SQL/management/inter-node ports, and Warehouse replication group.
3. Prepare packages and licenses, then choose a deployment method.
- [machclusterctl](./cluster-edition/#machclusterctl): configuration-based deployment and plan review
- [Manual Installation](./cluster-edition/#manual-machcoordinatoradmin): stepwise registration and startup
4. Check role-specific node states and replication configuration, then connect to a Broker with SQL.
5. Use the [Validation Checklist](./validation-checklist/) to check ingestion/query behavior
separately from replication state within a group.
A successful SQL connection does not establish that every replica is healthy. Different Warehouse
groups need not contain identical data. Continue with
[Cluster Operations](/dbms/operations-configuration-recovery/cluster/) for failure handling.
## Upgrade
For an existing system, follow [Upgrade](./upgrade/). Check compatibility of data files, SQL and
SDKs, configuration, licenses, and backup/recovery paths as well as the new executables. Do not
replace production settings with package samples or treat binary replacement alone as a completed
upgrade.
Keep baseline measurements and a backup from before the upgrade, and measure recovery time in an
isolated environment. Returning to an earlier version requires data and configuration that the
earlier version can read, not just its executables.
## Next Steps
Installation validation begins operational preparation. Use the
[Quick Start](/dbms/getting-started/quick-start/) to learn the SQL workflow, then model your data in
[Table Type Selection and Schema Design](../data-modeling-table-design/). Before production use,
prepare dedicated accounts, ingestion error handling, retention and backup, representative load
tests, and monitoring.
See [Observability and Diagnosis](/dbms/operations-configuration-recovery/diagnosis-observability/) and
[Performance Tuning Approach](/dbms/performance-tuning/performance-approach/) for that process.
---
title: "3.1 Pre-Installation Preparation"
url: https://docs.machbase.com/dbms/installation-deployment-upgrade/pre-install-preparation/
language: en
kind: page
---
# 3.1 Pre-Installation Preparation
Preparation establishes not only where to copy the software, but also where data will remain,
which account will run the server, and how clients will connect. Check package and operating
system compatibility, then prepare storage, networking, and licensing. Use the supplied release
information and support policy to determine supported configurations.
## Deployment Information to Decide First
| Item | Decision | Why it matters |
|---|---|---|
| Edition and version | Standard or Cluster, server and SDK versions | Select supported SQL features and deployment methods |
| OS account | Server account and file owner | Align access to configuration, data, and log directories |
| Installation home | Absolute path containing executables and configuration | Identify the instance controlled by administration commands |
| Data path | Actual `DBS_PATH`, file system, and free space | Identify data to preserve across restarts and upgrades |
| Connection details | Server address, SQL and management ports, allowed clients | Prevent port conflicts and connections to the wrong instance |
| Recovery and licensing | Backup storage, restoration procedure, and license | Establish recovery and operating limits |
The OS account `machbase` and database user `SYS` are different identities. The former controls
processes and file access; the latter controls SQL connections and database permissions.
A server can start successfully while loading or backup still fails because of file or SQL permissions.
| Topic | Contents |
|---|---|
| [Requirements](#pre-install-requirements) | OS compatibility, resources, and network ports |
| [Package Layout](#package) | Package naming, directories, and executables |
| [License Installation](#license) | Installing license.dat and checking its state |
## Pre-Installation Requirements
### Operating System and Package
Match the operating system and CPU architecture to the package. Supported OS releases and
minimum versions can change between releases; check the release information supplied with the
package instead of relying on a fixed version list.
### System Resources
CPU, memory, disk, and network requirements depend on ingestion rate, retention, indexes, and
ROLLUP. Include raw data, backups, and operational headroom as well as installation space, and
validate capacity and throughput with representative workloads.
Estimate raw row counts from the ingestion rate and retention period, then load representative
data to measure row size, compression, and index costs. Include replica storage for Cluster.
Separate directories on the same disk are neither independent failure domains nor independent
I/O devices.
### Default Port
| Port | Purpose |
|---|---|
| **5656** | SQL client connections over native TCP |
Set `PORT_NO` in `$MACHBASE_HOME/conf/machbase.conf` to change the SQL port.
The `MACHBASE_PORT_NO` environment variable is also used, so inspect the environment of the
shell or service that starts the server. Specify the new port in clients too. A changed
environment variable does not retroactively modify a running server.
Allow the required incoming connections through the firewall. Cluster also requires ports for
Coordinator links and administration, Brokers, Warehouses, and Deployers.
### Linux Kernel and Process Settings
Check the following before installation.
#### File Descriptor Limit
A low file descriptor limit can constrain workloads that open many files. Defaults depend on
the OS and account configuration; check the account that will run the server.
```bash
# Check the current limit
ulimit -Sn
```
This example uses 65535. If the current limit is lower, update `/etc/security/limits.conf`
and verify the result in a new login session.
```
* hard nofile 65535
* soft nofile 65535
```
Log in again as the server account. If a service manager starts the server, also check the
limit configured for that service.
```bash
ulimit -Sn
# Expected output: 65535
```
#### Port Reservation
Reserve Machbase service ports so the OS does not select them automatically as ephemeral ports.
This does not prevent another process from explicitly binding the same port.
```bash
current=$(cat /proc/sys/net/ipv4/ip_local_reserved_ports)
ports=5656
sudo sysctl -w net.ipv4.ip_local_reserved_ports="${current:+$current,}$ports"
```
Merge existing reserved ports rather than overwriting them. For persistence, merge the required
values into `net.ipv4.ip_local_reserved_ports` in `/etc/sysctl.conf`.
```
net.ipv4.ip_local_reserved_ports = 5656
```
### Time Synchronization
Use NTP or `chrony` to synchronize system clocks. Accurate time matters for time-series data,
and clocks across Cluster nodes must be synchronized.
```bash
# Check the time zone
ls -l /etc/localtime
date
```
## Package Layout
### Package Naming
Package names follow this pattern, with Edition-specific values.
```
machbase-EDITION-VERSION-OS-CPU-BIT-MODE.EXT
```
| Field | Meaning | Example |
|---|---|---|
| EDITION | Edition identifier | `SDK`, `cluster` |
| VERSION | Major.Minor.Fix.AUX | `8.7.0.official` |
| OS | Operating system | `LINUX`, `WINDOWS` |
| CPU | CPU architecture | `X86` |
| BIT | Architecture bit width | `64` |
| MODE | Build mode | `release` |
| EXT | File extension | `tgz` for Linux; `zip` or an installer for Windows |
The Standard Edition Linux archive uses the name `machbase-SDK-...tgz`.
- Standard: `machbase-SDK-8.7.0.official-LINUX-X86-64-release.tgz`
- Cluster: `machbase-cluster-8.7.0.official-LINUX-X86-64-release.tgz`
Different minor versions may differ in DB file or protocol compatibility. Check the destination
release's compatibility guidance and the [Upgrade Procedure](../upgrade/) for supported paths,
including fix-version changes.
### Installation Directories
Extracting the archive creates the following layout under `$MACHBASE_HOME`.
```text
$MACHBASE_HOME/
├── bin/ Executables
├── conf/ Configuration, including machbase.conf
├── dbs/ Data storage
├── doc/ License documents
├── include/ C/C++ headers
├── install/ Makefile include files
├── lib/ Shared libraries
├── package/ Additional Cluster packages
├── sample/ Examples
├── trc/ Server trace logs
├── tutorials/ Tutorials
├── utility/ Utilities
└── 3rd-party/ Grafana plugins and other integrations
```
### Main Executables
| Executable | Purpose |
|---|---|
| `machbased` | Server daemon |
| `machadmin` | Server start, stop, and database creation |
| `machsql` | SQL command-line client |
| `machloader` | Bulk file loading and export |
| `csvimport` | CSV import |
| `csvexport` | CSV export |
| `tagmetaimport` | Bulk TAG metadata registration |
Cluster packages also contain administration tools such as `machcoordinatoradmin` and
`machdeployeradmin`. `machclusterctl` is available in packages built to include it.
### Configuration Files
Edition-specific sample files are under `$MACHBASE_HOME/conf/`.
```bash
ls $MACHBASE_HOME/conf/
# machbase.conf
# machbase.conf.sample.standard
# machbase.conf.sample.edge
# machloader.conf.sample
```
The active configuration file is `machbase.conf`. The full Standard package includes a copy of
`machbase.conf.sample.standard` as `machbase.conf`. If your package does not include the active
file, copy the sample for the appropriate Edition and configure it.
Standard/Edge samples include `TRANSACTION_BUSY_TIMEOUT_MS`, `TRANSACTION_SYNCHRONOUS`, and
`TRANSACTION_JOURNAL_MODE` for TRANSACTION write contention and durability. Start with the
defaults, then evaluate concurrent writes and durability requirements before changing them.
## License Installation
Without a license file, the server uses the default `COMMUNITY` license information. Before
installation, confirm that its limits fit the planned features and capacity. If you need a
separate license, prepare it before the first startup. Successful startup alone does not
establish that the license meets production requirements.
### Check License State
Use `VIOLATE_STATUS` and `VIOLATE_MSG` in `V$LICENSE_INFO` to inspect the installed license
and any limit violations. Do not edit the license file's contents.
### Installation Methods
#### Method 1: Copy the File Before Startup
Copy `license.dat` to `$MACHBASE_HOME/conf/`. The server reads it when starting.
```bash
cp license.dat $MACHBASE_HOME/conf/license.dat
```
#### Method 2: Use machadmin
`machadmin` validates and installs the file. If the server is running, it also requests a
license reload.
```bash
machadmin -t /path/to/license.dat
```
#### Method 3: Use SQL on a Running Server
Run the following in `machsql`. The server process must be able to read the specified file.
```sql
ALTER SYSTEM INSTALL LICENSE = '/path/to/license.dat';
```
### Verify Installation
#### machadmin
```bash
machadmin -f
```
#### V$LICENSE_INFO
```sql
SELECT ID, ISSUE_DATE, TYPE, CUSTOMER, VIOLATE_STATUS, VIOLATE_MSG
FROM V$LICENSE_INFO;
```
Check that `VIOLATE_STATUS` is zero. You can also inspect license information in `machsql`:
```sql
SHOW LICENSE;
```
---
title: "3.2 Standard Edition Installation"
url: https://docs.machbase.com/dbms/installation-deployment-upgrade/standard-edition/
language: en
kind: page
---
# 3.2 Standard Edition Installation
Standard Edition handles SQL and storage on one server. Install in this order: prepare the
package, configure the execution environment, create the database, check the license, start
the server, and verify SQL. A single server is not limited to small data sets; measure whether
its resources meet your throughput and retention requirements.
## Installation Paths
| OS | Method | Procedure |
|---|---|---|
| Linux | Tarball (.tgz) | [Tarball Installation](#linux-tarball) |
| Linux | Docker container | [Docker Installation](#linux-docker) |
| Windows | ZIP or installer | [Windows Package Installation](#windows-package) |
First complete [Linux Preparation](#linux-preparation-environment-linux) or
[Windows Preparation](#windows-preparation-environment-windows).
## Linux Installation
| Method | Suitable situations |
|---|---|
| [Tarball](#linux-tarball) | Server deployments where you manage installation and data directories |
| [Docker](#linux-docker) | Development and test environments using containers |
Complete [Pre-Installation Preparation](../pre-install-preparation/) before tarball installation.
For Docker, prepare Docker Engine and the necessary volume and port permissions.
### Linux Preparation
File descriptor limits, clock synchronization, port reservation, and firewall configuration
are shared preparation tasks. Configure them for the server account and operating environment
using [Pre-Installation Preparation](../pre-install-preparation/).
### Tarball Installation
The following procedure extracts a Linux tarball to install Standard Edition.
#### 1. Create an OS User
Create the dedicated server account:
```bash
sudo useradd -m -d /home/machbase machbase
sudo passwd machbase
```
Log in as `machbase` for the remaining steps.
#### 2. Download and Extract the Package
The example assumes that the package is under `/home/machbase/packages/`.
Substitute the actual package name and extract it into a new installation directory without
an existing instance. For an existing installation, follow [Upgrade](../upgrade/).
```bash
machbase_package=/home/machbase/packages/machbase-SDK-8.7.0.official-LINUX-X86-64-release.tgz
test -r "$machbase_package" &&
mkdir /home/machbase/machbase_home &&
tar zxf "$machbase_package" -C /home/machbase/machbase_home &&
cd /home/machbase/machbase_home
```
Check the extracted layout:
```bash
ls -l
# bin/ conf/ dbs/ doc/ include/ lib/ trc/ ...
```
#### 3. Set Environment Variables
Add these variables to `~/.bashrc`:
```bash
export MACHBASE_HOME=/home/machbase/machbase_home
export PATH="$MACHBASE_HOME/bin:$PATH"
export LD_LIBRARY_PATH="$MACHBASE_HOME/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
```
Apply them to the current shell:
```bash
source ~/.bashrc
```
#### 4. Create the Database
`machadmin -c` creates the physical instance's database files. It is different from SQL
`CREATE DATABASE`, which adds a logical database. First verify `MACHBASE_HOME`,
`conf/machbase.conf`, and the actual `DBS_PATH`. If a database already exists, inspect the
target path instead of deleting and recreating it.
```bash
machadmin -c
# Database created successfully.
```
#### Check Configuration and Licensing Before Startup
Check `PORT_NO` and `DBS_PATH` in `conf/machbase.conf`. If you use a separate license,
install it by copying the file or using `machadmin -t`, then check it with `machadmin -f`
before starting. See [License Installation](../pre-install-preparation/#license).
#### 5. Start the Server
```bash
machadmin -u
# Machbase server started successfully.
```
Check server state:
```bash
machadmin -e
```
#### 6. Test the Connection
Connect with `machsql`. The initial administrator credentials are `SYS` / `MANAGER`;
use the current password if it has been changed.
```bash
machsql
# Machbase server address (Default:127.0.0.1) :
# Machbase user ID (Default:SYS)
# Machbase User Password :
# MACHBASE_CONNECT_MODE=INET, PORT=5656 EDITION=STANDARD
# Mach>
```
Run the basic SQL test:
```sql
CREATE LOG TABLE install_check (id INTEGER, val DOUBLE);
INSERT INTO install_check (id, val) VALUES (1, 3.14);
SELECT id, val FROM install_check;
DROP TABLE install_check;
```
Verify one row with `id=1` and `val=3.14`, followed by a successful DROP. If
`install_check` already exists, choose an unused practice name instead of deleting it.
#### Stop the Server
Run this only if you need to stop the server after verification.
```bash
machadmin -s
# Machbase server shut down successfully.
```
#### Change the Port
Change `PORT_NO` in `$MACHBASE_HOME/conf/machbase.conf` or set the environment variable:
```bash
export MACHBASE_PORT_NO=7878
```
Apply the setting while the server is stopped and start it again. This variable affects the
current shell; a service needs the same environment configured separately. Specify the changed
port when connecting, for example `machsql -s 127.0.0.1 -P 7878 -u SYS`.
### Docker Installation
Docker packages the server and its execution environment in a container. Even for development
and testing, prepare Docker Engine, storage volumes, ports, and file descriptor limits on the host.
The deployment example uses `machbase/machbase`. If you build your own image, replace it with
the local image name, such as `machbase:latest`.
#### Check the Image
```bash
docker pull machbase/machbase
docker image ls machbase/machbase
```
#### Create the Container
```bash
docker create \
--name machbase \
--ulimit nofile=65535 \
-p 5656:5656 \
-v /data/machbase:/home/machbase/machbase/dbs \
machbase/machbase
```
| Option | Meaning |
|---|---|
| `-p 5656:5656` | Host SQL port mapped to container SQL port |
| `--ulimit nofile=65535` | File descriptor limit available to the server |
| `-v /data/machbase:...` | Preserve the data directory on the host |
Data stored only in a container's writable layer is removed with the container. Confirm that
the actual data path is mounted on the volume, and also keep backups against volume deletion
or disk failure.
`docker create` does not start the server yet. The container's server account must be able to
write `/data/machbase`. If it contains existing DB files, verify their version and instance.
Check the image's actual version and `MACHBASE_HOME`, then pin a validated tag or digest for
production. Do not assume the untagged public image always contains 8.7.0.
If you use a separate license, copy it before starting:
```bash
docker cp /path/to/license.dat machbase:/home/machbase/machbase/conf/license.dat
```
Start the prepared container:
```bash
docker start machbase
```
#### Check Container State
```bash
docker ps
docker logs machbase
```
#### Test the Connection
##### machsql Inside the Container
```bash
docker exec -it machbase machsql
# Mach>
```
##### From the Host
If `machsql` is installed on the host:
```bash
machsql -s 127.0.0.1 -P 5656 -u SYS -p MANAGER
```
#### Stop and Restart the Container
```bash
docker stop machbase
docker start machbase
```
#### Update the License
Copy the replacement file into the running container and install it with the license command.
Configuration and licensing are separate from the data volume; retain their originals so
that they can be reapplied when recreating a container.
```bash
docker cp /path/to/license.dat machbase:/tmp/license.dat
docker exec machbase machadmin -t /tmp/license.dat
docker exec machbase machadmin -f
```
## Windows Installation
### Before You Start
- Check supported Windows versions in the supplied package's release information.
- Match package bit width and CPU architecture to the operating system.
Complete [Windows Preparation](#windows-preparation-environment-windows) first.
### Installation Methods
| Method | Description |
|---|---|
| [Windows Package Installation](#windows-package) | Installation wizard; creates environment variables and shortcuts |
### Windows Preparation
Check firewall configuration before installation.
#### Allow the SQL Port
| Port | Protocol | Purpose |
|---|---|---|
| 5656 | TCP | SQL client connections |
##### Configure Through the UI
1. Open **Control Panel → Windows Defender Firewall → Advanced settings**.
2. Select **Inbound Rules**, then **New Rule**.
3. Choose **Port**, then **Next**.
4. Choose **TCP**, enter `5656` in **Specific local ports**, then **Next**.
5. Choose **Allow the connection**, then **Next**.
6. Select only the network profiles where access is needed: **Domain**, **Private**, or **Public**.
7. Name the rule, for example `Machbase`, and select **Finish**.
Restrict the rule's remote-address scope to the actual clients. If remote access is unnecessary,
you do not need an inbound allow rule.
##### Configure with Administrator PowerShell
The following creates an inbound rule; apply the same profile and address restrictions.
```powershell
New-NetFirewallRule -DisplayName "Machbase SQL" -Direction Inbound -Protocol TCP -LocalPort 5656 -Action Allow
```
#### Visual C++ Redistributable
The runtime requires Visual C++ Redistributable. The installer may handle this dependency.
If it is missing, obtain the appropriate runtime from Microsoft's official site.
### Windows Package Installation
A ZIP contains `bin\`, `conf\`, `dbs\`, and `trc\` at its root.
The installer creates `machbase_home\` below the installation directory and configures
environment variables and shortcuts.
#### Installation Procedure
1. Download the Windows package.
2. For a ZIP, extract it into the desired installation directory.
```cmd
mkdir C:\machbase
tar -xf machbase-SDK-8.7.0.official-WINDOWS-X86-64-release.zip -C C:\machbase
```
3. If using an installer instead, run it and select **Next**.
4. Select the installation directory. Its default follows the form
`C:\machbase-\`. Change it if needed, then select **Next**.
5. After installation, select **Next → Close**.
#### Initialize a ZIP Installation
In Command Prompt, set the extracted directory as the installation home. This example uses
a new installation at `C:\machbase`. Check the active configuration and license first.
Run `-c` only for a new instance without existing DB files.
```cmd
set "MACHBASE_HOME=C:\machbase"
set "PATH=%MACHBASE_HOME%\bin;%PATH%"
machadmin.exe -c
machadmin.exe -f
machadmin.exe -u
machadmin.exe -e
```
These `set` commands affect the current window. Use the same installation home and executable
path in later windows or services. If an installer already created the database, do not repeat
the ZIP creation command.
#### Start and Stop the Server
An installer provides these desktop or Start-menu shortcuts:
- **start Machbase** runs `machadmin.exe -u`.
- **stop Machbase** runs `machadmin.exe -s`.
- **machsql** opens the SQL console.
#### Connect from Command Prompt
For an installer, `\machbase_home\bin` is added to `PATH`.
For a ZIP, add its `bin\` directory yourself or run the executable by its full path.
```cmd
machsql -s 127.0.0.1 -P 5656 -u SYS -p MANAGER
```
The initial credentials are `SYS` / `MANAGER`.
#### Installation Layout
A ZIP places `bin\`, `conf\`, `dbs\`, and `trc\` directly below the extraction directory.
An installer places the same layout under `machbase_home\`.
See [Package Layout](../pre-install-preparation/#package).
---
title: "3.3 Cluster Edition Installation and Deployment"
url: https://docs.machbase.com/dbms/installation-deployment-upgrade/cluster-edition/
language: en
kind: page
---
# 3.3 Cluster Edition Installation and Deployment
Cluster Edition separates SQL connections, storage, replication, and node management into
different roles. Before installation, assign each role to a host and choose its ports and
storage paths. Ordinary SQL connections go to Brokers; administration commands go to the
appropriate management node.
## Node Roles
| Node | Role |
|---|---|
| Coordinator | Manage cluster metadata and monitor node state |
| Deployer | Deploy packages and initialize nodes |
| Lookup | Process reference data and related requests |
| Broker | Accept client SQL, parse it, and distribute queries |
| Warehouse | Store data and execute queries |
The YAML example below uses three hosts with two Coordinators, three Deployers, two Lookup
nodes (one master and one monitor), two Brokers, and two Warehouses in one replication group.
Choose actual node counts and placement according to availability, throughput, and the
capacity that must remain after a failure.
## Deployment Methods
| Method | Description | When to use it |
|---|---|---|
| [machclusterctl](#machclusterctl) | Automated deployment from `cluster.yaml` | Recommended starting point for new installations |
| [Manual administration](#manual-machcoordinatoradmin) | Register and deploy nodes through Coordinator commands | When individual steps require direct control |
## Installation Order
1. Understand the [Cluster Configuration](#overview).
2. Complete [Environment Preparation](#preparation-environment-cluster-edition), including SSH,
kernel settings, and clock synchronization.
3. Prepare packages, paths, and the [License](../pre-install-preparation/#license) installation method.
4. Deploy, start, and verify licensing using the chosen method.
5. Complete [Installation Validation](../validation-checklist/).
## Cluster Configuration Overview
Coordinator, Deployer, Lookup, Broker, and Warehouse have separate responsibilities.
Understand their relationships before planning deployment.
### Node Roles in Detail
#### Coordinator
The Coordinator manages metadata, node registration, and monitoring. Consider Primary/Secondary
redundancy. Although management and running data-processing paths are separate, Coordinator
failure alone does not establish whether all SQL will continue. Check other nodes and client
impact and follow a tested recovery procedure.
- Configuration: `$MACHBASE_COORDINATOR_HOME/conf/machbase.conf`
- Tool: `machcoordinatoradmin`
- Main ports: `CLUSTER_LINK_PORT_NO`, `HTTP_ADMIN_PORT`
The defaults are `CLUSTER_LINK_PORT_NO=3868` and `HTTP_ADMIN_PORT=5779`. This chapter
explicitly uses `5101` and `5102` for the Coordinator link and management ports to avoid conflicts.
#### Deployer
The Deployer carries out package deployment and initialization requested by the Coordinator.
Place one on each node host or operate it on a separate deployment server.
- Tool: `machdeployeradmin`
#### Lookup
Lookup nodes process reference data. Their configured roles are `master`, `monitor`, or
`slave`.
#### Broker
Brokers accept and parse client SQL and distribute work to Warehouses. Ordinary applications
connect to Broker addresses. Direct Warehouse connections are for management diagnostics,
such as comparing replicas, and are separate from application connection paths. Redundant Brokers
are recommended.
- Default client connection port: 5656
#### Warehouse
Warehouses store data and execute queries. Members of the same group replicate data for
availability. At least two members per group are recommended.
### Configuration Illustration
```
[Client]
│ (SQL, 5656)
▼
[Broker ×2] ──────────────────────────────────────────
│ (Query distribution)
├─► [Warehouse group1-node1] ◄──Replication──► [Warehouse group1-node2]
└─► [Warehouse group2-node1] ◄──Replication──► [Warehouse group2-node2]
[Coordinator Primary] ◄──HA──► [Coordinator Secondary]
│ (Metadata and node monitoring)
[Deployer]
│
[Lookup master / monitor]
```
### Edition Comparison
See
[Standard and Cluster Differences](../../core-concepts/concepts-edition/#differences-standard-edition-cluster).
## Prepare the Cluster Environment
Prepare the following on the relevant hosts before deployment.
### File Descriptor Limit
Edit the limit configuration on each node host:
```bash
sudo vi /etc/security/limits.conf
```
```
* hard nofile 65535
* soft nofile 65535
```
Check the result in a new login session as the server account. If a service manager starts
the process, also check that service's file descriptor limit.
```bash
ulimit -Sn
# 65535
```
### Create OS Users
Create the `machbase` account on each host:
```bash
sudo useradd -m machbase --home-dir /home/machbase
sudo passwd machbase
```
### SSH Key Authentication
For `machclusterctl`, the deployment host must be able to connect to all target hosts using
noninteractive key-based SSH.
```bash
# Create a key on the deployment host; skip if one already exists
ssh-keygen -t rsa -b 4096
# Register the public key on each target host
ssh-copy-id machbase@192.168.1.11
```
The address is an example; repeat registration and verification for every target host.
Check that the connection works without a password:
```bash
ssh machbase@192.168.1.11 'hostname'
```
### Network Kernel Parameters
These are tuning examples, not required values for every server. First measure current
kernel settings, memory use, and network bottlenecks. Compare memory and latency as well as
throughput after changes, and persist only changes that have been validated.
```bash
sudo sysctl -w net.core.rmem_default=33554432
sudo sysctl -w net.core.wmem_default=33554432
sudo sysctl -w net.core.rmem_max=268435456
sudo sysctl -w net.core.wmem_max=268435456
sudo sysctl -w 'net.ipv4.tcp_rmem=262144 33554432 268435456'
sudo sysctl -w 'net.ipv4.tcp_wmem=262144 33554432 268435456'
sudo sysctl -w 'net.ipv4.tcp_mem=8388608 8388608 8388608'
```
To make validated settings persistent, add them to `/etc/sysctl.conf`.
### Time Synchronization
Synchronize all node clocks with NTP or `chrony`.
```bash
# Example with chrony
sudo systemctl enable chronyd
sudo systemctl start chronyd
chronyc tracking
```
An isolated installation environment without a time server can set its initial clock manually.
Replace this illustrative timestamp with the actual current time. A manual setting does not
continuously correct clock drift; establish clock synchronization before production.
```bash
sudo date -s "2025-01-02 12:34:56"
```
### Reserve Ports
Reserve the ports used by Machbase on each host.
```bash
current=$(cat /proc/sys/net/ipv4/ip_local_reserved_ports)
ports=5101-5110,5201-5202,5301-5302,5401,5500-5503,5656
sudo sysctl -w net.ipv4.ip_local_reserved_ports="${current:+$current,}$ports"
```
Merge existing reservations rather than overwriting them. Adjust the ranges for the actual
configuration, including cluster links, Coordinator/Deployer management, service ports, and
Warehouse replication-manager ports.
## Deploy with machclusterctl
`machclusterctl` deploys and manages a cluster from `cluster.yaml`. It uses SSH to deploy
packages, initialize nodes, and coordinate startup and shutdown.
### Prerequisites
- Run commands on the host that will contain the Primary Coordinator. The package and SSH
private key must be readable there.
- Configure key-based SSH from that host to every target host.
- Ensure permission to create and write the parent directories of each node's `home_path`
and `dbs_path`.
- If a separate license is required, prepare the package and licensing procedure before automatic
startup. Do not invent a license property in the YAML.
- Complete [Cluster Environment Preparation](#preparation-environment-cluster-edition).
### Workflow
| Step | Document |
|---|---|
| 1. Write cluster.yaml | [YAML Configuration](#machclusterctl-cluster-yaml) |
| 2. Validate it | [YAML Validation](#machclusterctl-validation-yaml) |
| 3. Install and start | [Initial Installation](#machclusterctl-initial) |
| 4. Inspect state | [Status Checks](#machclusterctl-status-check-state) |
| 5. Make later configuration changes | [Cluster Operations](../../operations-configuration-recovery/cluster/) |
| 6. Diagnose a failure | [Cluster Troubleshooting](../../troubleshooting/cluster/) |
### Write cluster.yaml
`cluster.yaml` declares the nodes and their paths. Replace the example addresses with actual hosts.
`origin_path` is the archive read on the Primary Coordinator host where the command runs.
`home_path` and `dbs_path` belong to the node's host. `deployer` references the Deployer
that will deploy and control that node.
#### Example File
```yaml
version: "1"
cluster:
name: mc-prod
hosts:
node1:
address: machbase@192.168.1.10
node2:
address: machbase@192.168.1.11
node3:
address: machbase@192.168.1.12
package:
name: machbase
origin_path: /home/machbase/packages/machbase-cluster-8.7.0.official-LINUX-X86-64-release.tgz
ssh:
key_file: /home/machbase/.ssh/id_rsa
defaults:
coordinator:
home_path: /home/machbase/coordinator
cluster_link_port: 5101
http_admin_port: 5102
deployer:
home_path: /home/machbase/deployer
cluster_link_port: 5201
http_admin_port: 5202
lookup:
home_path: /home/machbase/lookup
cluster_link_port: 5301
broker:
home_path: /home/machbase/broker
cluster_link_port: 5401
service_port: 5656
warehouse:
home_path: /home/machbase/warehouse
cluster_link_port: 5501
service_port: 5500
coordinators:
- alias: coord-primary-1
host: node1
role: primary
- alias: coord-secondary-1
host: node2
role: secondary
deployers:
- alias: deployer-1
host: node1
- alias: deployer-2
host: node2
- alias: deployer-3
host: node3
lookup:
- alias: lookup-master-1
host: node1
deployer: deployer-1
type: master
- alias: lookup-monitor-1
host: node2
deployer: deployer-2
type: monitor
brokers:
- alias: broker-1
host: node1
deployer: deployer-1
dbs_path: /data/machbase/broker-1/dbs
- alias: broker-2
host: node2
deployer: deployer-2
warehouse_groups:
- name: group1
nodes:
- alias: warehouse-group1-1
host: node2
deployer: deployer-2
dbs_path: /data/machbase/warehouse-group1-1/dbs
- alias: warehouse-group1-2
host: node3
deployer: deployer-3
dbs_path: /data/machbase/warehouse-group1-2/dbs
```
#### Field Reference
| Field | Meaning |
|---|---|
| `version` | YAML schema version; currently `"1"` |
| `cluster.name` | Cluster identifier, also used for `destroy` confirmation |
| `cluster.hosts` | Host aliases and SSH `address` values in `user@host` form |
| `cluster.package.name` | Package name registered with the Coordinator |
| `cluster.package.origin_path` | Input archive for `install`, `apply`, and `upgrade` |
| `cluster.package.registered_path` | Observed Coordinator package-store path recorded by `export`; not an input archive |
| `cluster.ssh.key_file` | SSH private-key path; no password field is used |
| `cluster.defaults` | Role-specific `home_path`, `cluster_link_port`, and `service_port` defaults |
| `cluster.coordinators` | Coordinator nodes; `role` is `primary` or `secondary` |
| `cluster.deployers` | Deployer nodes |
| `cluster.lookup` | Lookup nodes; `type` is `master`, `monitor`, or `slave` |
| `cluster.brokers` | Broker nodes; clients connect to `service_port` |
| `cluster.warehouse_groups` | Warehouse groups and their members |
#### Redundancy and Placement
Recommended configuration:
- Two Coordinators: Primary and Secondary for high availability.
- One or more Deployers.
- One Lookup `master` and at least one `monitor`.
- Two or more Brokers for load balancing.
- Two Warehouses per group for replication and high availability.
If several nodes of the same type share one host, explicitly give the additional nodes
different `home_path` values and ports. Coordinator and Deployer management use `http_admin_port`.
Broker and Warehouse nodes can specify `dbs_path`; it is a path on the node's host. If omitted,
the normal `DBS_PATH` behavior of `machcoordinatoradmin --add-node` applies.
The legacy `cluster.package.path` remains a compatibility input, but new YAML should use
`origin_path`. Only Coordinator and Deployer use `http_admin_port`; do not configure HTTP
ports for Broker, Lookup, or Warehouse.
Validate the file after editing.
### Validate the YAML
`validate` statically checks syntax, required values, aliases, port conflicts, and topology.
#### Validation Command
```bash
machclusterctl validate -f cluster.yaml
```
#### Checks
| Item | Validation |
|---|---|
| YAML syntax | Parsing errors |
| Environment substitution | `${VAR}` and `${VAR:-default}` expressions |
| Required fields | Cluster, hosts, package, and required node values |
| Aliases | Duplicate node aliases |
| Ports | Declared conflicts on the same host |
| Topology | Primary Coordinator, Lookup master/monitor, and Deployer references |
#### Example Output
```text
Validation passed.
```
Correct reported errors and validate again.
#### Inspect the Installation Plan
Use `install --dry-run --verbose` to inspect SSH access, package availability, remote directory
permissions, and the proposed actions.
```bash
machclusterctl install -f cluster.yaml --dry-run --verbose
```
After installation, `apply --dry-run` compares current state with the YAML and displays the
change plan without applying remote changes.
```bash
machclusterctl apply -f cluster.yaml --dry-run --verbose
```
#### Common Errors
| Error | Cause | Correction |
|---|---|---|
| `field ... not found` | Unsupported YAML key | Use a supported `cluster.*` field |
| `required field ...` | Missing value | Add the reported field |
| `duplicate alias` | Reused alias | Assign unique node aliases |
| `port conflict` | Same port on the same host | Change the conflicting port; changing only the home path does not resolve it |
| `deployer ... not found` | Unknown Deployer reference | Set `deployer` to a configured Deployer alias or host:port |
### Initial Installation
After preparing and validating the YAML, inspect the plan before installing.
#### 1. Install the Cluster
```bash
machclusterctl install -f cluster.yaml --dry-run --verbose
```
If the checks pass, perform the installation:
```bash
machclusterctl install -f cluster.yaml --yes --verbose
```
The command automatically performs these steps:
1. Copies and extracts packages on each node.
2. Generates each node's `machbase.conf` and configures ports.
3. Initializes the Coordinator database.
4. Registers each node with the Coordinator.
#### 2. Start the Cluster
`install` prepares and starts the Coordinator, Deployers, Lookup, Brokers, and Warehouses.
Use the following only when you need to start the installed cluster again:
```bash
machclusterctl start
```
#### 3. Check State
```bash
export MACHBASE_COORDINATOR_HOME=/home/machbase/coordinator
machclusterctl status
```
To explicitly select a Coordinator home when several are managed from one host:
```bash
machclusterctl status --coordinator /home/machbase/coordinator
```
Output follows `machcoordinatoradmin --cluster-status-full --verbose`. The following is a
partial illustration of the columns, not the complete node list from the YAML. Coordinator
and Broker can show role states such as `primary` or `leader`. Compare registered node counts
and role-specific desired and actual states, rather than requiring every row to say `normal`.
```
+-------------+--------------------------------+--------------------------------+--------------------------------+-------------------------------+-------------+-----------------+----------+
| Node Type | Node Name | Group Name | Group State | Desired & Actual State | RP State | Disk(%) (00/00) | Ping(μs) |
+-------------+--------------------------------+--------------------------------+--------------------------------+-------------------------------+-------------+-----------------+----------+
| coordinator | coord-1(192.168.1.10:5101) | Coordinator | normal | primary | primary | ----------- | --------------- | 214 |
| deployer | deployer-1(192.168.1.10:5201) | Deployer | normal | running | running | ----------- | --------------- | 100 |
| broker | broker-1(192.168.1.11:5401) | Broker | normal | leader | leader | ----------- | --------------- | 100 |
| warehouse | wh-g1-1(192.168.1.13:5501) | group1 | normal | normal | normal | running | 26.9 | 100 |
+-------------+--------------------------------+--------------------------------+--------------------------------+-------------------------------+-------------+-----------------+----------+
```
#### 4. Test Client Connections
Connect to a Broker IP and its SQL port:
```bash
machsql -s 192.168.1.11 -P 5656 -u SYS -p MANAGER
# Mach>
```
#### Stop the Cluster
```bash
machclusterctl stop
```
### After Installation
For topology changes, node addition/removal, and recovery, follow
[Cluster Operations](../../operations-configuration-recovery/cluster/).
For failure classification and recovery decisions, see
[Cluster Troubleshooting](../../troubleshooting/cluster/).
## Deploy Manually with machcoordinatoradmin
If `machclusterctl` is unavailable, prepare Coordinator and Deployer processes directly,
then register packages and nodes. Do not repeat creation commands in an already deployed
cluster. The manual example is separate from the YAML example above.
Run each command on the host for its corresponding role.
| Host | Roles |
|---|---|
| `192.168.1.10` | Primary Coordinator, Deployer, Lookup master |
| `192.168.1.11` | Deployer, Lookup monitor, Broker |
| `192.168.1.13` | Deployer, first Warehouse in group1 |
| `192.168.1.14` | Deployer, replica Warehouse in group1 |
| `192.168.1.20` | Optional Secondary Coordinator |
Separate homes and ports for roles on the same host. Prepare a Deployer on each host containing
data-processing nodes and match the node's `--deployer` address to that host.
### Manual Deployment Order
1. Read [Package Preparation](#manual-machcoordinatoradmin-package) and prepare full and lightweight
packages.
2. Install and start [Coordinator and Deployers](#manual-machcoordinatoradmin-coordinator-deployer).
3. [Register the lightweight package](#manual-machcoordinatoradmin-package) with the running Coordinator.
4. Register and start [Lookup, Broker, and Warehouse](#manual-machcoordinatoradmin-lookup-broker-warehouse).
5. [Check overall state](#manual-machcoordinatoradmin-lookup-broker-warehouse) after registration and startup.
### Comparison with machclusterctl
| Item | machclusterctl | Manual deployment |
|---|---|---|
| Configuration | One `cluster.yaml` | Edit each node's `machbase.conf` |
| Package deployment | Automated remote copy | Direct Coordinator/Deployer installation; Deployer installs data-processing nodes |
| Node registration | Automatic | `machcoordinatoradmin --add-node` |
| Start and stop | `machclusterctl start/stop` | Individual node commands |
Manual deployment offers direct control but requires tracking every host and role.
For new installations, start with `machclusterctl` when available.
### Package Preparation and Registration
Install the full package on Coordinator and Deployer hosts. Register the lightweight package
used for Broker and Warehouse deployment only after the Coordinator is running.
#### Package Types
| Package | Target | Contents |
|---|---|---|
| Full | Coordinator and Deployer | All executables |
| Lightweight | Broker and Warehouse | Only data-processing files; smaller package |
Example filenames:
- Full: `machbase-cluster-8.7.0.official-LINUX-X86-64-release.tgz`
- Lightweight: `machbase-cluster-8.7.0.official-LINUX-X86-64-release-lightweight.tgz`
#### Install Full Packages
##### Coordinator Host
```bash
# Run on the Coordinator host
mkdir -p /home/machbase/coordinator
scp machbase@package-host:/path/to/machbase-cluster-8.7.0.official-LINUX-X86-64-release.tgz /home/machbase/
tar zxf /home/machbase/machbase-cluster-8.7.0.official-LINUX-X86-64-release.tgz -C /home/machbase/coordinator
```
##### Each Deployer Host
```bash
mkdir -p /home/machbase/deployer
scp machbase@package-host:/path/to/machbase-cluster-8.7.0.official-LINUX-X86-64-release.tgz /home/machbase/
tar zxf /home/machbase/machbase-cluster-8.7.0.official-LINUX-X86-64-release.tgz -C /home/machbase/deployer
```
Do not manually extract the lightweight package into Broker or Warehouse homes here.
After it is registered, the Deployer selected by `--add-node` deploys it to `--home-path`.
#### Register the Package with the Coordinator
Install Coordinator and Deployer first, then register the package with the running Coordinator:
```bash
$MACHBASE_COORDINATOR_HOME/bin/machcoordinatoradmin --add-package=machbase \
--file-name="/home/machbase/machbase-cluster-8.7.0.official-LINUX-X86-64-release-lightweight.tgz"
```
Broker and Warehouse registration later references it as `--package-name=machbase`.
#### Set Role-Specific Environments
Replace `package-host` and `/path/to/` with the actual package location.
Use separate management shells for Coordinator and Deployer even when they share a host.
Apply these settings to the appropriate shell and use the same values for services or login
initialization files.
```bash
# Coordinator management shell
export MACHBASE_COORDINATOR_HOME=/home/machbase/coordinator
export MACHBASE_HOME=$MACHBASE_COORDINATOR_HOME
export PATH=$MACHBASE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$MACHBASE_HOME/lib:$LD_LIBRARY_PATH
```
```bash
# Separate Deployer management shell
export MACHBASE_DEPLOYER_HOME=/home/machbase/deployer
export MACHBASE_HOME=$MACHBASE_DEPLOYER_HOME
export PATH="$MACHBASE_HOME/bin:$PATH"
export LD_LIBRARY_PATH="$MACHBASE_HOME/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
```
### Install Coordinator and Deployers
Prepare the packages first, then start the Coordinator before registering Deployers.
#### Coordinator
##### 1. Configure machbase.conf
Edit `$MACHBASE_COORDINATOR_HOME/conf/machbase.conf`:
```bash
vi $MACHBASE_COORDINATOR_HOME/conf/machbase.conf
```
Use this host's address and the planned ports:
```
CLUSTER_LINK_HOST = 192.168.1.10 # This host's IP
CLUSTER_LINK_PORT_NO = 5101
HTTP_ADMIN_PORT = 5102
```
##### 2. Create Metadata and Start
```bash
machcoordinatoradmin -c
machcoordinatoradmin -u
```
##### 3. Register the Coordinator Itself
```bash
machcoordinatoradmin --add-node="192.168.1.10:5101" \
--node-type=coordinator \
--http-admin-port=5102
```
##### 4. Verify Registration
```bash
machcoordinatoradmin --cluster-status
```
#### Optional Secondary Coordinator
For redundancy, prepare the Secondary package and configuration on its own host.
Register it from the **Primary Coordinator first**, then start it on the Secondary host.
```bash
# Register from the Primary first
machcoordinatoradmin --add-node="192.168.1.20:5101" \
--node-type=coordinator \
--http-admin-port=5102
# Then start on the Secondary host, specifying the Primary
machcoordinatoradmin -u --primary=192.168.1.10:5101
```
Complete Primary registration before starting the Secondary.
#### Deployer
Perform these steps on every Deployer host in the manual layout. Replace
`CLUSTER_LINK_HOST` with that host's own IP. Copying another host's address would make the
communication address inconsistent with the actual process location.
##### 1. Configure machbase.conf
```
CLUSTER_LINK_HOST = 192.168.1.10 # Deployer host IP
CLUSTER_LINK_PORT_NO = 5201
HTTP_ADMIN_PORT = 5202
```
##### 2. Start the Deployer
```bash
machdeployeradmin -c
machdeployeradmin -u
```
##### 3. Register Deployers with the Coordinator
After all Deployers are running, use the Primary Coordinator's management shell:
```bash
machcoordinatoradmin --add-node="192.168.1.10:5201" \
--node-type=deployer \
--http-admin-port=5202
machcoordinatoradmin --add-node="192.168.1.11:5201" \
--node-type=deployer --http-admin-port=5202
machcoordinatoradmin --add-node="192.168.1.13:5201" \
--node-type=deployer --http-admin-port=5202
machcoordinatoradmin --add-node="192.168.1.14:5201" \
--node-type=deployer --http-admin-port=5202
```
### Install Lookup, Broker, and Warehouse
Prepare Coordinator and Deployers first. Register the lightweight package with the Coordinator
with `--add-package` before adding Broker or Warehouse nodes.
#### Lookup Nodes
Register and start the master and monitor. This example places them on the Primary and Broker
hosts, using the local Deployer on each. Confirm that the homes are not used by existing Lookup
nodes.
```bash
machcoordinatoradmin --add-node="192.168.1.10:5301" \
--node-type=lookup \
--lookup-type=master \
--deployer="192.168.1.10:5201" \
--home-path="/home/machbase/lookup"
machcoordinatoradmin --add-node="192.168.1.11:5301" \
--node-type=lookup \
--lookup-type=monitor \
--deployer="192.168.1.11:5201" \
--home-path="/home/machbase/lookup"
machcoordinatoradmin --startup-node="192.168.1.10:5301"
machcoordinatoradmin --startup-node="192.168.1.11:5301"
```
#### Broker
##### 1. Check Registration Settings
The node configuration is generated during `--add-node` and deployed through the Deployer. Choose
the cluster link and
SQL service ports before registration. Do not configure an HTTP management port for Broker.
```
CLUSTER_LINK_HOST = 192.168.1.11 # Broker host IP
CLUSTER_LINK_PORT_NO = 5401
PORT_NO = 5656 # Client SQL port
```
##### 2. Register from the Coordinator
```bash
machcoordinatoradmin --add-node="192.168.1.11:5401" \
--node-type=broker \
--deployer="192.168.1.11:5201" \
--package-name=machbase \
--home-path="/home/machbase/broker" \
--dbs-path="/data/machbase/broker_dbs" \
--port-no=5656
```
| Parameter | Meaning |
|---|---|
| `--add-node` | Node IP:CLUSTER_LINK_PORT_NO |
| `--node-type` | `broker`, `warehouse`, or `lookup` |
| `--deployer` | IP:CLUSTER_LINK_PORT_NO of the Deployer that installs and controls the node |
| `--package-name` | Registered package name |
| `--home-path` | Node installation home |
| `--dbs-path` | Broker/Warehouse data path; otherwise uses the default `DBS_PATH` |
| `--port-no` | SQL or node service port |
| `--replication` | Warehouse replication-manager address in `host:port` form |
##### 3. Start the Node
From the Coordinator:
```bash
machcoordinatoradmin --startup-node="192.168.1.11:5401"
```
#### Warehouses
Warehouse nodes belong to groups. Nodes in the same group replicate data.
##### 1. Check Registration Settings
The configuration is generated during registration and deployed through the Deployer. Establish the
cluster link,
service port, and replication-manager address beforehand.
```
CLUSTER_LINK_HOST = 192.168.1.13
CLUSTER_LINK_PORT_NO = 5501
PORT_NO = 5500
```
##### 2. Register the Nodes
```bash
machcoordinatoradmin --add-node="192.168.1.13:5501" \
--node-type=warehouse \
--deployer="192.168.1.13:5201" \
--package-name=machbase \
--home-path="/home/machbase/warehouse_g1_1" \
--dbs-path="/data/machbase/warehouse_g1_1_dbs" \
--port-no=5500 \
--replication=192.168.1.13:5502 \
--group=group1 \
--no-replicate
machcoordinatoradmin --add-node="192.168.1.14:5501" \
--node-type=warehouse \
--deployer="192.168.1.14:5201" \
--package-name=machbase \
--home-path="/home/machbase/warehouse_g1_2" \
--dbs-path="/data/machbase/warehouse_g1_2_dbs" \
--port-no=5500 \
--replication=192.168.1.14:5502 \
--group=group1
```
There is no separate `--add-group` step. Specify the Warehouse group with `--group` when
registering each node.
##### 3. Start the Nodes
```bash
machcoordinatoradmin --startup-node="192.168.1.13:5501"
machcoordinatoradmin --startup-node="192.168.1.14:5501"
```
#### Check All Nodes
```bash
machcoordinatoradmin --cluster-status
```
Compare role-specific states: Coordinator may be `primary`, Broker `leader`, and Warehouses
`normal`, `sync-active`, or `sync-standby`. Verify the configured node list rather than
one universal state string.
#### Verify the First Connection
Connect to the Broker SQL port and run `SELECT CURRENT_DATABASE();` and a representative query.
For subsequent node control and recovery, use
[Cluster Operations](../../operations-configuration-recovery/cluster/) and
[Cluster Troubleshooting](../../troubleshooting/cluster/).
---
title: "3.4 Upgrade"
url: https://docs.machbase.com/dbms/installation-deployment-upgrade/upgrade/
language: en
kind: page
---
# 3.4 Upgrade
An upgrade is more than replacing executables: it verifies that existing data, configuration, and
applications retain their intended behavior on the new version. First confirm a supported version
transition, prepare a restorable backup, and define service-resumption criteria. Adjust the example
8.7.0 package names and paths to the distribution you received.
## Before upgrading
- Check compatibility between the current and target versions. A minor-version change can involve
a different database file format.
- Take a backup before upgrading. See [backup procedures](/dbms/operations-configuration-recovery/backup-restore-mount/#backup).
- Identify clients with ongoing INSERT or Append work.
### Machbase 8.7.0 pre-upgrade check
When upgrading from 8.5 to 8.7.0, identify the following dependencies and migrate them to supported
alternatives before replacing binaries.
1. Locate removed properties `HTTP_AUTH`, `HTTP_ENABLE`, `HTTP_MAX_MEM`, `HTTP_PORT_NO`,
`RS_CACHE_*`, `STREAM_THREAD_COUNT`, and `STREAM_WAIT_MS` in the current configuration and
compare them with the supported list. Remove obsolete properties and preserve supported ones.
In particular, Cluster `HTTP_ADMIN_PORT` remains a management port; do not remove it simply
because it matches `HTTP_*`.
2. Migrate applications calling `/machbase` or `/machiot` to a backend using a supported SDK.
3. Update SQL or operational scripts that invoke `STREAM_*` procedures or `FLUSH RESULT_CACHE`.
4. Migrate C/C++ applications using `machcli.h` and `MachCLI*()` to Machbase SQLCLI or ODBC.
SQLCLI and ODBC are separate API sets.
5. Replace operational workflows and dashboards that depend on WebAdmin/MWA with command-line
tools or separate applications.
See
[version compatibility](/dbms/reference/support-scope-constraints/compatibility-version/#removed-features-870)
for the complete list of removed and retained features.
## Upgrade paths
| Edition | Method | Procedure |
|---------|--------|-----------|
| Standard Edition | Stop the server and replace the package | [Standard upgrade](#standard-edition) |
| Cluster Edition | Restart Brokers and Warehouses sequentially | [Online upgrade](#cluster-edition-online) |
| Cluster Edition | Stop the entire cluster | [Full-stop upgrade](#cluster-edition-full-stop) |
For Cluster Edition, choose online or full-stop upgrade according to availability requirements.
---
## Standard Edition upgrade
Stop the server, replace the package, and restart. Apply this procedure only to supported version
transitions that can open the existing physical database files. If the transition requires data
conversion or export/import, follow the release-specific migration procedure first.
### Preparation
1. **Take and validate a backup.** Use a supported BACKUP command and verify restoration in a
separate environment. Simply copying a live data directory does not establish a recoverable backup.
2. **Finish client activity.** Complete ongoing Append and INSERT operations and close connections.
3. **Record the current version:**
```bash
machbased -v
```
### Procedure
#### 1. Stop the server
```bash
machadmin -s
# Machbase server shut down successfully.
```
#### 2. Preserve the existing package and configuration
Retain the current configuration and license as well as executables and libraries in a separate
location. Check that the destinations below do not already contain an earlier backup.
```bash
cp -a "$MACHBASE_HOME/bin" "$MACHBASE_HOME/bin.bak"
cp -a "$MACHBASE_HOME/lib" "$MACHBASE_HOME/lib.bak"
cp -a "$MACHBASE_HOME/conf" "$MACHBASE_HOME/conf.bak"
```
Preserve `dbs/` and data under any separately configured `DBS_PATH`. These copies preserve
executables and configuration; they do not replace a database backup. Once the new version has
modified data, do not assume that restoring old binaries alone can recover the instance.
Use the backup restoration path you have validated.
#### 3. Extract and apply the new package
Extract the new package into a separate working directory and inspect its contents and configuration
changes first.
```bash
upgrade_stage=$(mktemp -d)
tar zxf machbase-SDK-8.7.0.official-LINUX-X86-64-release.tgz -C "$upgrade_stage"
```
Extraction alone does not replace executables in the existing installation. The following example
applies executables, libraries, and headers from a Standard tarball containing `bin/`, `lib/`,
and `include/`. Confirm that the server is stopped first. If any command fails, do not proceed
to server startup.
```bash
(
set -e
test -n "$MACHBASE_HOME"
test -x "$upgrade_stage/bin/machbased"
test -d "$upgrade_stage/lib"
test -d "$upgrade_stage/include"
test -d "$MACHBASE_HOME/bin"
test -d "$MACHBASE_HOME/lib"
test -d "$MACHBASE_HOME/include"
cp -a "$upgrade_stage/bin/." "$MACHBASE_HOME/bin/"
cp -a "$upgrade_stage/lib/." "$MACHBASE_HOME/lib/"
cp -a "$upgrade_stage/include/." "$MACHBASE_HOME/include/"
"$MACHBASE_HOME/bin/machbased" -v
)
```
Preserve the existing `conf/machbase.conf`, license, and data at the actual `DBS_PATH`, and merge
new configuration entries into the existing configuration. Copying replaces distribution files
with matching names but does not automatically remove files present only in the old version.
Select SDK and plugin files explicitly for the new version, and consult the release instructions
for additional replacements and removals. Start the server only after checking the reported binary
version and reviewing the configuration.
#### 4. Start the server
```bash
machadmin -u
# Machbase server started successfully.
```
#### 5. Check the version
```bash
machbased -v
# Connect to the server.
machsql -s 127.0.0.1 -P 5656 -u SYS -p MANAGER
```
```sql
SELECT EDITION, BINARY_DB_MAJOR_VERSION, BINARY_DB_MINOR_VERSION FROM V$VERSION;
```
### Important constraints
- Do not delete `dbs/` or initialize it with `machadmin -d`.
- Minor-version upgrades may require database file migration. Read the release notes.
- On Windows, stop the Machbase service before applying the new package or installer.
---
## Cluster Edition upgrade
Choose a method according to the required service interruption.
| Method | Service impact | Suitable situation |
|--------|----------------|--------------------|
| [Online upgrade](#cluster-edition-online) | Sequential Broker/Warehouse restarts | Replacing only Brokers and Warehouses |
| [Full-stop upgrade](#cluster-edition-full-stop) | Cluster outage | A maintenance window is available, or a major-version transition is required |
### Common precautions
- Do not run DDL or DELETE during the upgrade.
- Do not concurrently add, start, stop, or remove nodes.
- Online upgrade targets Brokers and Warehouses. Use full-stop upgrade to replace Coordinators,
Deployers, or Lookups as well.
- Take a backup before upgrading.
---
### Online upgrade
Upgrade Brokers and Warehouses sequentially while the cluster is running. If all binaries,
including Coordinator, Deployer, and Lookup, must be replaced, use
[full-stop upgrade](#cluster-edition-full-stop).
#### Procedure
##### 1. Change the package in cluster.yaml
Set `cluster.package.name` and `cluster.package.origin_path` to the new package. When package
contents change, assign distinct names to both the package and archive file.
```yaml
cluster:
package:
name: machbase-v8.7.0
origin_path: /home/machbase/packages/machbase-cluster-8.7.0.official-LINUX-X86-64-release.tgz
```
`registered_path` is the Coordinator package-repository path recorded by `machclusterctl export`.
Use `origin_path` to specify the upgrade archive.
##### 2. Inspect the execution plan
```bash
machclusterctl upgrade -f cluster.yaml --online --dry-run --verbose
```
##### 3. Run the online upgrade
```bash
machclusterctl upgrade -f cluster.yaml --online --yes --verbose
```
Omitting `--online` also selects online mode, but specifying it makes the operational procedure
explicit.
##### 4. Check overall status
```bash
machclusterctl status
```
#### Manual upgrade reference
When calling `machcoordinatoradmin --upgrade-node` directly, specify both the target node and the
package name.
```bash
machcoordinatoradmin --upgrade-node=192.168.1.11:5401 --package-name=machbase-v8.7.0
```
Limit online targets to Brokers and Warehouses. If only one Broker remains, upgrading it can
interrupt client connections for that period.
Online mode avoids stopping the entire cluster; it is not an HA-aware rolling upgrade that
guarantees uninterrupted service. Warehouse groups may temporarily become read-only, so validate
application reconnection, retry handling, and write delays. Use full-stop mode if protocol
compatibility changes or binaries for every role must be aligned.
Check the role-specific states and versions of all target nodes, then validate representative
queries, input, and replication.
---
### Full-stop upgrade
Stop the entire cluster and upgrade all nodes. Use this method when all binaries, including
Coordinator, Deployer, and Lookup, must be replaced or when the database file format changes.
#### Procedure
##### 1. Finish client activity
Verify that all INSERT, Append, and SELECT work has completed.
##### 2. Change the package in cluster.yaml
Set `cluster.package.name` and `cluster.package.origin_path` to the new package. There must be no
pending topology changes, such as node additions, removals, or port changes. Apply topology changes
with `apply` before running the upgrade.
`machclusterctl upgrade --full-stop` replaces the package in every node home, including
Coordinator and Deployer. Set `origin_path` to a complete Cluster package containing
`machcoordinatoradmin` and `machdeployeradmin`.
##### 3. Inspect the execution plan
```bash
machclusterctl upgrade -f cluster.yaml --full-stop --dry-run --verbose
```
##### 4. Run the full-stop upgrade
```bash
machclusterctl upgrade -f cluster.yaml --full-stop --yes --verbose
```
With the entire cluster stopped, `machclusterctl` extracts the package into a temporary staging
location and applies it to the node homes.
#### Manual deployment reference
For manual deployment, register the new package with the Coordinator.
```bash
machcoordinatoradmin --add-package=machbase-v8.7.0 \
--file-name=/home/machbase/packages/machbase-cluster-8.7.0.official-LINUX-X86-64-release.tgz
```
Stop nodes in this order: Warehouse → Broker → Lookup → Deployer → Coordinator.
```bash
machcoordinatoradmin --shutdown-node=192.168.1.13:5501
machcoordinatoradmin --shutdown-node=192.168.1.14:5501
machcoordinatoradmin --shutdown-node=192.168.1.11:5401
machcoordinatoradmin --shutdown-node=192.168.1.10:5301
machdeployeradmin --shutdown
machcoordinatoradmin --shutdown
```
When replacing packages in node homes, preserve `conf/machbase.conf` and the `dbs/`, `meta/`,
and `package/` directories. Extract the package in a separate working location, then replace
files while excluding the paths that must be retained.
Start nodes in this order: Coordinator → Deployer → Lookup → Broker → Warehouse.
```bash
machcoordinatoradmin --startup
machdeployeradmin --startup
machcoordinatoradmin --startup-node=192.168.1.10:5301
machcoordinatoradmin --startup-node=192.168.1.11:5401
machcoordinatoradmin --startup-node=192.168.1.13:5501
machcoordinatoradmin --startup-node=192.168.1.14:5501
```
After restarting, synchronize Broker and Warehouse package metadata with the new package name.
```bash
machcoordinatoradmin --upgrade-node=192.168.1.11:5401 --package-name=machbase-v8.7.0
machcoordinatoradmin --upgrade-node=192.168.1.13:5501 --package-name=machbase-v8.7.0
machcoordinatoradmin --upgrade-node=192.168.1.14:5501 --package-name=machbase-v8.7.0
```
##### 5. Check status
```bash
machclusterctl status
```
Before resuming service, verify Broker connections, representative reads and writes, replication,
license, and configuration as well as node status. Compare
[installation validation](../validation-checklist/) with the pre-upgrade record. Retain the previous
package, configuration, and backup until verification is complete.
#### Important constraints
- A major-version upgrade may change the database file format. Read the release notes and take a
backup before upgrading.
- Do not delete or initialize `conf/machbase.conf`, `dbs/`, `meta/`, or `package/`.
---
title: "3.5 Installation Validation Checklist"
url: https://docs.machbase.com/dbms/installation-deployment-upgrade/validation-checklist/
language: en
kind: page
---
# 3.5 Installation Validation Checklist
Validation goes beyond checking that a process is running. Actual clients must connect to the
intended server version and configuration, then write and read data within their permissions.
Work through server status, connectivity, version and license, SQL, and Cluster replication in order.
Record the execution host, installation home, connection address and port, time, and any errors for
each result. For an upgrade, compare these with the pre-upgrade record. `SYS`/`MANAGER` are initial
exercise credentials; substitute the actual account. Check that exercise table names do not collide
with existing business objects.
## Standard Edition checklist
### 1. Check the server process
```bash
machadmin -e
# Machbase server is running with PID().
```
You can also inspect processes directly, but distinguish the target server from servers running
under other installation homes. Verify that `MACHBASE_HOME` identifies the instance being checked.
```bash
ps -ef | grep machbased | grep -v grep
```
### 2. Check the listening port
```bash
ss -tlnp | grep 5656
# LISTEN 0 128 0.0.0.0:5656 ...
```
Use a port-inspection tool appropriate for the operating system. Replace `5656` with the actual SQL
port and check that the listener uses the intended interface. A listening socket and a successful
remote client connection are separate checks; test connectivity from the application host too.
### 3. Test a machsql connection
```bash
machsql -s 127.0.0.1 -P 5656 -u SYS -p MANAGER
# MACHBASE_CONNECT_MODE=INET, PORT=5656 EDITION=STANDARD
# Mach>
```
If local connections succeed but remote connections fail, check the address, port, listener, and
firewall. For authentication errors, check the username, authentication method, and expiry status.
If connection succeeds but object access fails, check the current database and SQL permissions
separately.
### 4. Check the version
```sql
SELECT * FROM V$VERSION;
SELECT CURRENT_DATABASE();
```
### 5. Check the license
```sql
SELECT ID, ISSUE_DATE, TYPE, VIOLATE_STATUS FROM V$LICENSE_INFO;
```
Check that `VIOLATE_STATUS` is 0 and that the installed license type and validity period match the
operating plan. If a violation is reported, inspect `VIOLATE_MSG` and the server logs for its cause.
### 6. Test basic SQL
```sql
CREATE LOG TABLE check_test (id INTEGER, ts DATETIME);
INSERT INTO check_test (id, ts) VALUES (1, NOW);
SELECT id, ts FROM check_test;
DROP TABLE check_test;
```
Verify that one row with `id=1` and a timestamp is returned and that DROP succeeds. This checks
only basic LOG input. If the service uses TAG, TRANSACTION, Append, or other paths, also test
representative writes and queries using those tables and the actual SDK.
## Additional Cluster Edition checks
### 7. Check cluster node status
```bash
machcoordinatoradmin --cluster-status
# Compare node count and role-specific states with the deployment record.
```
### 8. Test a Broker connection
```bash
machsql -s 192.168.1.11 -P 5656 -u SYS -p MANAGER
```
```sql
SELECT * FROM V$NODE_STATUS;
```
Healthy state names differ by role: for example, Coordinator `primary`, Broker `leader`, and
Warehouse replication states. Do not compare every node against one status string. Check the
desired and actual states, group membership, and addresses against the deployment record.
### 9. Check data replication
First verify input and queries through a Broker, then inspect replication within Warehouse groups.
Reading a row through a Broker does not by itself prove that every replica has synchronized.
```sql
-- Insert through a Broker.
CREATE LOG TABLE cluster_check_test (id INTEGER, ts DATETIME);
INSERT INTO cluster_check_test (id, ts) VALUES (1, NOW);
-- Verify the input through the Broker.
SELECT COUNT(*) FROM cluster_check_test;
```
Direct SQL connections to Warehouses are an administrative replication diagnostic, not the normal
application path. Use `machcoordinatoradmin --cluster-status` to identify active and standby peers
in the same replication group. When needed, connect to each peer's native port with an
administrative account and compare the same query results. Do not assume nodes in different
Warehouse groups all contain the same rows.
After validation, remove the exercise table through the Broker connection.
```sql
DROP TABLE cluster_check_test;
```
---
## Acceptance criteria and troubleshooting
Proceed with service acceptance once server, connection, permission, representative SQL, and
required replication checks pass. To validate persistence, retain test data in a separate
validation environment and compare results before and after a normal restart. Do not reinitialize
a server containing business data merely to check installation.
If a check fails, preserve the first error and relevant logs, then investigate the failed stage.
- Server log: `$MACHBASE_HOME/trc/machbase.trc`
- See [operations and diagnostics](/dbms/operations-configuration-recovery/diagnosis-observability/).
---
title: "4. Table Types and Schema Design"
url: https://docs.machbase.com/dbms/data-modeling-table-design/
language: en
kind: section
---
# 4. Table Types and Schema Design
This chapter translates the meaning of your data and its mutation/query requirements into
Machbase DBMS tables and columns. Once the database is running, decide what one row represents.
Choosing a familiar table type first and forcing all data into it can create conflicting
requirements for history, updates, and retention.
If table roles are new to you, first read [Data Model Concepts](/dbms/core-concepts/concepts/).
This chapter connects those concepts to concrete schemas and designs you can validate.
## Start with the Data
Before choosing a table name, describe one row in a sentence. “One measurement from one sensor,”
“one equipment state-change event,” and “the current installation information for one device”
describe different units of data. Records from the same equipment can still have different
storage roles.
| Design question | Decision to make |
|---|---|
| What does one row represent? | A measurement, event, current state, or reference record |
| How is its subject identified? | Tag names, business keys, and duplicate-ingestion identifiers |
| What does the time or axis mean? | Measurement time, receipt time, or distance/position |
| How are values interpreted? | Units, type ranges, precision, NULL, and missing observations |
| Which mutations are needed? | Appends, historical corrections, key changes, and deletion boundaries |
| Which queries recur? | Tag/time ranges, predicate searches, key lookups, aggregates, and joins |
| What must be retained? | Raw data, aggregates, reference history, and state recoverable after restart |
| What can be undone after failure? | Table/API transaction scope, retries, and reconstruction |
Data size or input frequency alone does not determine the table type. Check that values grouped
in a row describe the same observation or event, and design common identifiers for records that
must be joined. Having a join key does not mean the DBMS automatically enforces every business
relationship. Check the constraints supported by the selected table type.
## Contents
| Order | Section | Design outcome |
|-----:|---|---|
| 4.1 | [Choose a Table Type](./table-types-selection-type/) | A type matching mutation, query, persistence, and Edition requirements |
| 4.2 | [Schema Objects](./schema-objects-definition/) | Columns and types, identifiers, defaults/constraints, indexes, and views |
| 4.3 | [Data Mutation Policy](./alter-data-mutation-policy/) | Correction/deletion boundaries and failure handling |
| 4.4 | [Anti-Patterns](./table-types-patterns-type-anti/) | Reasons to revise choices that do not match requirements |
| 4.5 | [Modeling Patterns](./patterns-modeling/) | Concrete combinations of history, state, and reference data |
Choose a type in 4.1 and define structure and mutation rules in 4.2–4.3. Check the anti-patterns in
4.4, then adapt the patterns in 4.5 to your data. Comparison tables are starting points; follow
the linked table guides and references for exact SQL and supported conditions.
## An Equipment-Monitoring Example
Temperature, alarms, equipment definitions, and display state need not use the same storage model.
| Data | Meaning of one row | Storage role to consider |
|---|---|---|
| Temperature history | One sensor reading at a particular time | TAG for tag/time-range queries and aggregation |
| Alarm history | One alarm event at a particular time | LOG for append-oriented events |
| Equipment reference data | One device's name, location, and thresholds | LOOKUP, or TRANSACTION when multiple changes must share a transaction |
| Current display state | Recent state that can be recalculated | VOLATILE when authoritative data and a rebuild path exist |
This is an example, not a fixed prescription. A record containing several measurements, or different
mutation and retention requirements, may call for another design. Check Standard Edition support
when considering TRANSACTION.
Joining temperature history to current equipment definitions interprets that history using today's
names and locations. If you need the location or threshold at measurement time, retain it with
the history or model changes to the reference data. When maintaining a current-state cache, do not
assume the history insert and cache update form one transaction. Prepare a way to rebuild the cache
after failure.
## Validate with a Small Data Set
Before scaling up, use representative records and common queries to check the design.
1. Insert normal values together with NULLs, missing observations, boundary values, and duplicate
or late-arriving records.
2. Check that raw queries and aggregates use the same time basis, units, and NULL policy.
3. Compare results after corrections, deletion, and reingestion, including effects on related
rows and aggregates.
4. Distinguish data that must survive restart from state that must be recreated, and verify the
recovery sequence.
5. Increase ingestion rate, query range, and concurrency while measuring throughput, latency,
and storage.
Design for input failures, applications using old schemas, and restart recovery alongside the
normal workflow. Before evolving a schema, inspect existing data and dependencies in views,
ROLLUPs, and clients. When needed, migrate to a new table, validate it, and then switch over.
Continue with the [Table Usage Chapters](/dbms/) and
[Development and Application Integration](/dbms/development-tools-integration/) for implementation,
and [Performance Tuning](/dbms/performance-tuning/) for measurement.
---
title: "4.1 Choose a Table Type"
url: https://docs.machbase.com/dbms/data-modeling-table-design/table-types-selection-type/
language: en
kind: page
---
# 4.1 Choose a Table Type
Choose a table type that matches the data early in design. An unsuitable type can reduce
performance and limit required functionality.
- **[Selection Guide](/dbms/data-modeling-table-design/table-types-selection-type/#selection-decision)**
- **[Type Comparison](/dbms/data-modeling-table-design/table-types-selection-type/#comparison-tag-log-rdb-volatile-lookup)**
- **[TRANSACTION vs LOOKUP](/dbms/data-modeling-table-design/table-types-selection-type/#comparison-rdb-vs-lookup)**
For table roles and storage concepts, see [Data Model Concepts](../../core-concepts/concepts/#time-series).
The same data can need different types depending on whether you accumulate history or update
current state. Review mutation, query, and persistence requirements together.
### Describe the data before selecting a type
Before naming a table, describe the fact represented by one row in a sentence. Even for the
same equipment, one temperature reading, one current operating state, and one maintenance
job have different row semantics, keys, and update patterns.
| Design question | Equipment monitoring decision |
|---|---|
| What does one row represent? | One sensor reading or an equipment's current state |
| How is it found? | Sensor name and event time, equipment ID, or maintenance job number |
| How do values change? | Append history, correct errors, or overwrite current rows |
| Must changes commit together? | Whether job registration and part quantity changes share a transaction |
| How long is it retained? | Raw and aggregate retention periods; whether state is rebuildable after restart |
| What is the scale? | Tag count, rows/s, row size, memory used by reference data and indexes |
For example, consider TAG for temperature history, LOG for alarms, and LOOKUP for equipment
code tables. If inventory changes and job registration must commit together, consider
TRANSACTION in Standard Edition. A current-state cache can use VOLATILE if it is rebuildable
from source data. Align row semantics and failure recovery before combining these records
in one table.
## Selection guide
Use the following flow to narrow candidates, then verify DML, transactions, memory usage,
and edition support in the comparison tables. Even reference data may need TRANSACTION
instead of LOOKUP when multiple changes must share one transaction.
### Decision flow
```text
Is the data sensor/device measurements?
├── YES → Time axis? YES → TAG TABLE (BASETIME)
│ Distance axis? YES → TAG TABLE (BASEDISTANCE)
└── NO ↓
Is the data events/logs/packets? (append-only)
├── YES → LOG TABLE
└── NO ↓
Is the data a code table/reference data? (repeated reads and updates)
├── YES → LOOKUP TABLE
└── NO ↓
Is the data in-memory state/cache that can be discarded on server restart?
├── YES → VOLATILE TABLE
└── NO ↓
General relational business data (UPDATE/DELETE/SELECT/INSERT all required)
└── TRANSACTION TABLE
```
### Key criteria
| Question | Type |
|------|------|
| Time- or distance-based measurements? | TAG |
| Append raw events and remove only older ranges? | LOG |
| Reference data requiring a PRIMARY KEY and repeated reads/updates? | LOOKUP |
| Can the data be lost on server restart? | VOLATILE |
| General relational work (INSERT/UPDATE/DELETE/SELECT)? | TRANSACTION |
### Considerations
- Using a unique tag name for every event causes continuing growth in tags and metadata.
Consider LOG for events without recurring measurement targets.
- LOG cannot UPDATE or DELETE with general predicates, so it is unsuitable for mutable data.
Use retention-oriented `BEFORE`, `OLDEST`, or `EXCEPT` DELETE.
- TRANSACTION is Standard Edition only. In Cluster Edition, consider LOOKUP for small datasets
or an external RDBMS.
- VOLATILE data is lost on server restart.
## Type comparison
### Feature comparison
| Item | TAG | LOG | TRANSACTION | VOLATILE | LOOKUP |
|------|-----|-----|-----|----------|--------|
| DDL | `CREATE TAG TABLE` | `CREATE LOG TABLE` | `CREATE TABLE` / `CREATE TRANSACTION TABLE` / `CREATE TXN TABLE` | `CREATE VOLATILE TABLE` | `CREATE LOOKUP TABLE` |
| Main use | Sensors and measurements | Events and logs | Relational business data | Temporary aggregates | Codes and reference data |
| INSERT | Yes | Yes | Yes | Yes | Yes |
| UPDATE | Yes (Standard, tag/BASETIME predicates) | No | Yes | Yes | Yes |
| DELETE | Yes (BEFORE/predicates/all) | Yes (BEFORE/OLDEST/EXCEPT/all) | Yes | Yes (PK equality/all) | Yes (general predicates/all) |
| PRIMARY KEY | Required | No | Optional | Optional | Required |
| BASETIME | Required (time axis) | No | No | No | No |
| _arrival_time | No | Added automatically | No | No | No |
| Indexes | Tag/axis access, supported secondary indexes | BITMAP/KEYWORD/LSM | BTREE PK + secondary indexes | Key and secondary indexes | Key and secondary indexes |
| Persistence | Yes | Yes | Yes | No (memory) | Yes |
| Cluster Edition | Yes | Yes | No | Yes | Yes |
Append API support depends on the SDK and ingestion path as well as the table type. Check
your driver/table combination in the
[SDK Append Matrix](../../development-tools-integration/sdk-support-scope/#append-table-type-matrix).
### Storage characteristics
| Item | TAG | LOG | TRANSACTION | VOLATILE | LOOKUP |
|------|-----|-----|-----|----------|--------|
| Storage | Columnar | Columnar | Row-oriented (relational) | In memory | Persistent storage + all rows resident in memory |
| Capacity criteria | Tags, raw data, ROLLUP, retention | Raw data, search indexes, retention | Rows, indexes, transaction load | Memory for all rows and indexes | Memory for all rows and indexes; reload time after restart |
Small in-memory tables do not have a fixed row-count definition. Measure actual memory
including row width, variable-length values, and secondary indexes. Compression ratios and
server specifications alone cannot guarantee disk-table throughput either; run representative
ingestion and queries together.
### TRANSACTION table restrictions
TRANSACTION tables have these restrictions:
- **No Cluster Edition support:** Standard Edition only.
- **Minimum columns:** At least one.
## TRANSACTION vs LOOKUP
TRANSACTION and LOOKUP both store relational data, but differ in target scale and features.
### Comparison
| Item | TRANSACTION | LOOKUP |
|------|-----------|--------------|
| DDL | `CREATE TRANSACTION TABLE` | `CREATE LOOKUP TABLE` |
| PRIMARY KEY | Optional | Required |
| INSERT | Yes | Yes |
| UPDATE (with WHERE) | Yes | Yes |
| UPDATE (without WHERE) | Yes (all rows) | No |
| DELETE | Yes | Yes |
| Explicit transactions | Control multiple statements with COMMIT/ROLLBACK | Does not participate; changes are per statement |
| Indexes | BTREE PK + secondary indexes | In-memory key and secondary indexes |
| Data scale | Validate disk capacity and transaction load | Validate reference-data read/update load |
| JOIN target | Yes | Yes |
| Cluster Edition | No | Yes |
### Selection criteria
**Choose TRANSACTION for:**
- Data requiring explicit transactions and relational DML
- General relational workloads requiring UPDATE, DELETE, INSERT, and SELECT
- Queries on varied column combinations without a PRIMARY KEY
- Standard Edition environments
**Choose LOOKUP for:**
- Code tables and reference data
- PRIMARY KEY lookups and single-row UPDATE/DELETE
- Use in Cluster Edition
- Real-time reference-data updates
### Example
```sql
-- LOOKUP: country codes (PK lookup and predicate-based UPDATE)
CREATE LOOKUP TABLE country_code (
code VARCHAR(4) PRIMARY KEY,
name VARCHAR(64)
);
INSERT INTO country_code VALUES ('KR', 'Republic of Korea');
UPDATE country_code SET name = 'Korea' WHERE code = 'KR';
SELECT code, name FROM country_code WHERE code = 'KR';
-- TRANSACTION: order history (large scale, general UPDATE/DELETE)
CREATE TRANSACTION TABLE order_history (
order_id LONG PRIMARY KEY,
item_id INTEGER,
qty INTEGER,
amount DECIMAL(18,2)
);
INSERT INTO order_history VALUES (12345, 501, 1, 12000.00);
UPDATE order_history SET qty = 10 WHERE order_id = 12345;
SELECT order_id, qty, amount FROM order_history WHERE order_id = 12345;
DELETE FROM order_history WHERE order_id = 12345;
SELECT COUNT(*) FROM order_history WHERE order_id = 12345;
```
The first SELECT returns the updated country name; the order SELECT returns quantity `10`.
The final COUNT is `0`. Here, `amount` is an illustrative value maintained separately from
quantity and is not recalculated automatically. In an actual order model, define the
relationship among unit price, quantity, and total, and which columns change together.
When finished, use `DROP TABLE` to remove only the tables created in this example.
---
title: "4.2 Schema Object Definitions"
url: https://docs.machbase.com/dbms/data-modeling-table-design/schema-objects-definition/
language: en
kind: page
---
# 4.2 Schema Object Definitions
This page covers decisions for designing tables, columns, indexes, and views. Use the
[SQL Syntax Reference](/dbms/reference/sql/syntax/) as the authoritative source for syntax
and options rather than duplicating them here.
- **[Create and drop tables](#create-delete)**
- **[Alter tables](#alter)**
- **[Choose columns and data types](#selection-type-column-data-types)**
- **[Constraints and defaults](#constraints-defaults-condition)**
- **[Index design](#index-create-delete)**
- **[View design](#create-view)**
## Create and drop tables
First complete [Table Type Selection](../table-types-selection-type/). Then define table
names, columns, keys, constraints, indexes, and views for the selected type.
### Row granularity and column roles
Keep row granularity consistent within a table. Mixing daily equipment summaries and
second-level raw data as equivalent rows makes `COUNT` and `AVG` hard to interpret. Define
clear row granularity and query names for raw data and aggregates separately.
| Column role | Example | Design principle |
|---|---|---|
| Target identity | `sensor_id`, `equipment_id` | Use stable values separate from display names |
| Event timing | `measured_at`, `event_time` | Specify whether this is event time or arrival time |
| Measurement | `temperature_c`, `pressure_kpa` | Define units, valid ranges, and correction methods |
| Quality | `quality_code` | Distinguish missing values, measurement failures, and valid zero |
| Reference attributes | Location, equipment type | Decide between tag metadata and a separate reference table |
An existing business identifier, such as an external equipment code, is a natural key; a
separately assigned number is a surrogate key. If codes can change or be reused by different
sources, define their identity scope or consider a surrogate key. An auto-increment number
provides generation order; it does not automatically guarantee event time or duplicate-free
ingestion. A TAG name identifies a tag, not an individual measurement row.
Use table names that begin with a letter and contain letters, digits, and underscores. Avoid
reserved words and names that resemble system objects. Before dropping a table, check
dependent views, indexes, ROLLUPs, and retention policies.
For creation examples by type, see:
- [Create TAG Tables](/dbms/tag-table-usage/create-alter-drop/)
- [Create LOG Tables](/dbms/log-table-usage/create-alter-drop/)
- [TRANSACTION Tables](/dbms/rdb-table-usage/)
- [LOOKUP Tables](/dbms/lookup-table-usage/)
- [VOLATILE Tables](/dbms/volatile-table-usage/)
## Alter tables
Support for adding, dropping, renaming, and changing column types depends on the table type
and whether data exists. Before changing a production table, proceed in this order:
1. Check that the edition and table type support the intended `ALTER TABLE` operation.
2. Check dependencies on existing data, indexes, views, and application column order.
3. Measure execution time and locking effects with production-equivalent schemas and data volume.
4. If rollback is difficult, create a new table, validate it, and then switch over.
After adding a column, query both existing and newly inserted rows to check NULL and DEFAULT
behavior. Not all table types backfill existing rows with DEFAULT. Existing VOLATILE rows
and automatically registered TAG metadata rows, for example, have separate rules. Check the
actual type's DDL contract, including
[ARRAY DEFAULT Rules](/dbms/reference/sql/types/array/#default와-기존-row).
Also define application deployment and schema change order. Specify column lists where the
ingestion path supports them, and compare positional Append and binding code with the new
schema. For migration to a new table, compare per-key counts, time ranges, NULL rates, and
representative aggregates as well as total rows. Define how to prevent missing or duplicate
rows arriving during the switch.
For exact support and syntax, see the [ALTER TABLE Reference](/dbms/reference/sql/syntax/)
and [Support by Table Type](/dbms/reference/support-scope-constraints/table-types-type/).
## Choose columns and data types
Choose the smallest suitable type based on actual value ranges and operations. Distinguish
display formats from storage types.
| Data | Types to consider | Considerations |
| --- | --- | --- |
| Integer measurements and codes | `SHORT`, `INTEGER`, `LONG` families | Check reserved NULL values and ranges |
| Floating-point measurements | `FLOAT`, `DOUBLE` | Check precision, aggregate error, and reserved NULL values |
| Values requiring exact decimal arithmetic | `DECIMAL` | Define precision, scale, and rounding rules first |
| Timestamps | `DATETIME` | Design representable ranges and time zones with client/session policies |
| Short strings | `VARCHAR` | Check maximum length and encoding |
| Long text | `TEXT` | Check table support, sorting/aggregation restrictions, and index cost |
| Network addresses | `IPV4`, `IPV6` | Consider address types instead of strings |
| Structured documents | `JSON` | Check size limits, column promotion criteria, and table support |
| Binary data | `BINARY` | Variable or fixed length depends on table type |
| Fixed-size numeric collections | Numeric `ARRAY` | Distinguish element type/length, element NULLs, and whole-array NULL |
`VARCHAR(n)` measures bytes, which may differ from character counts for Korean text or emoji.
Integer types reserve some boundary values for NULL, so do not assume general-purpose
programming-language integer ranges. FLOAT and DOUBLE also recognize their positive maximum
values as NULL. Valid ranges and operation semantics take priority over choosing a small type.
### Exact decimal values: DECIMAL
Use `DECIMAL` when decimal accuracy matters, such as monetary amounts, tax rates, and
settlement values. Use `FLOAT` or `DOUBLE` for measurements that tolerate approximation and
need a wide exponent range. The distinction is semantic, not just storage size: choose
DECIMAL when rounded results directly drive business rules.
Define the following first:
| Decision | Rule |
|---|---|
| precision | Total significant digits, 1–65 |
| scale | Fractional digits, 0–30; cannot exceed precision |
| Omitted parameters | `DECIMAL` means `DECIMAL(10,0)`; `DECIMAL(M)` means `DECIMAL(M,0)` |
Input with more fractional digits than scale is rounded to the nearest value, with ties
away from zero. Rounding is final at storage time, so verify that it matches business rules.
Values exceeding precision produce an error rather than being truncated or converted to
floating point. Allow sufficient digits.
`SUM`, `AVG`, `MIN`, `MAX`, `GROUP BY`, `ORDER BY`, and `DISTINCT` use exact processing.
Operations without an exact DECIMAL path, such as percentiles or advanced statistics, convert
to DOUBLE and return approximate results. Distinguish exact aggregates from informational statistics.
Passing a value through floating point in the application loses accuracy regardless of the
storage type. Use decimal representations or strings: JDBC `BigDecimal`, Python
`decimal.Decimal`, or ODBC `SQL_NUMERIC`. For declarations, indexes, and client mappings, see
[DECIMAL and NUMERIC Fixed-Point Types](/dbms/reference/sql/types/decimal-numeric-fixed-point/).
### Structured documents: JSON
Use `JSON` for supplementary attributes whose keys vary by source or grow over time; fields
can be added without schema changes. Promote values frequently used in `WHERE` or `GROUP BY`
to separate columns. JSON columns cannot be primary keys, and LOOKUP does not support JSON
path indexes.
Account for size limits: a document is at most 32,768 bytes, and a JSON path is at most
512 bytes. Use JSON for attributes needed by queries rather than entire raw payloads.
| Table type | JSON column | Considerations |
|---|:---:|---|
| TAG, LOG, TRANSACTION | Yes | JSON functions and path queries supported |
| LOOKUP | Yes | Supported as an ordinary column; JSON path indexes unsupported |
| VOLATILE | No | JSON columns cannot be created |
Use `->` and `JSON_EXTRACT_*` for queries, and the `JSON_SET` family for changes. Mutation
functions are useful only when the table type supports `UPDATE`; for tables without row
updates, such as LOG or TAG, construct the document at ingestion time. For function support,
see [JSON Support by Table Type](/dbms/reference/sql/types/table-types-type-support-scope-json/).
### Restrictions to check before choosing a type
| Type | Design considerations |
|---|---|
| `TEXT` | Supported only in LOG and Standard Edition TRANSACTION. LOG TEXT columns cannot be used in `ORDER BY` or `GROUP BY`, or converted to VARCHAR with `MODIFY COLUMN`. Store sortable or aggregatable values separately as VARCHAR or numeric columns. |
| `BINARY` | LOG supports variable-length values up to 64MB; TAG uses fixed-length `BINARY(n)` of 1–32,767 bytes. LOOKUP and VOLATILE do not support it. |
| `DATETIME` | Represents 1970-01-01 through 2262-04-11 with nanosecond precision. Do not use arbitrary far-future dates to mean expiration or no expiration. |
| `ARRAY` | Fixed-length, one-dimensional numeric arrays with cardinality 1–1024. Distinguish whole-array NULL from element NULL. |
For complete ranges and table support, use the [Data Type Reference](/dbms/reference/sql/types/).
## Constraints and defaults
`PRIMARY KEY`, `NOT NULL`, and `DEFAULT` support differs by table type. A TAG `PRIMARY KEY`
identifies a tag; LOOKUP, VOLATILE, and TRANSACTION `PRIMARY KEY` columns determine row identity
and update access paths.
NULL represents an unknown or absent value, unlike zero or an empty time range. Replacing
measurement failures with DEFAULT 0 can distort averages and validity checks. `COUNT(*)`
counts rows; `COUNT(value)` counts rows where that column is non-NULL. Distinguish them when
reporting sample counts.
Defaults define behavior for omitted inputs; they do not replace validation. Validate
constraints unsupported by a table in ingestion or business applications. Do not assume
support for constraints such as foreign keys merely because another DBMS provides them.
Check [TRANSACTION Support](/dbms/reference/support-scope-constraints/rdb/).
Do not use system columns such as `_ARRIVAL_TIME` or `_RID` as application business keys.
Reference them only for documented query semantics, without depending on their storage
structure or generation mechanism.
## Index design
Indexes reduce query costs but add ingestion and storage costs.
First list representative queries and check predicate selectivity, the proportion of matching
rows. A monthly average across all equipment differs from a specific order lookup for one
device. Instead of indexing every filter or join column, compare `EXPLAIN` and actual
execution times, then retain useful indexes.
- Do not start with extra indexes when tag and time-range access is sufficient for TAG queries.
- For frequently filtered LOG columns, measure plans and selectivity before adding indexes.
- Design LOOKUP, VOLATILE, and TRANSACTION indexes around key lookups and joins.
- For word searches in long text, consider `KEYWORD` indexes supported by the table type.
For creation/deletion syntax and supported types, see the
[Index SQL Reference](/dbms/reference/sql/syntax/).
## View design
A view names a reusable query but does not store its results. Avoid unintended fixed time
ranges or unnecessary all-column reads in view definitions. Check dependent views before
changing base tables or columns.
Views can consistently expose frequently used columns and unit conversions. Creating a
view alone does not copy data or reduce query cost. A view joining historical events to
current reference data can change historical results when the reference data changes.
For attributes as they were at event time, use versioned reference data or attributes
recorded with the original event.
For view creation, querying, deletion, and restrictions, see the
[VIEW SQL Reference](/dbms/reference/sql/syntax/).
---
title: "4.3 Data Mutation Policy"
url: https://docs.machbase.com/dbms/data-modeling-table-design/alter-data-mutation-policy/
language: en
kind: page
---
# 4.3 Data Mutation Policy
Support for `UPDATE`, `DELETE`, and `TRUNCATE` differs by table type. This page summarizes
policies relevant to data model selection. Use the linked SQL references for exact syntax
and restrictions.
A mutation policy defines who may change which data and what can be rolled back on failure,
not just whether a value is editable. Treat current-state updates, raw measurement corrections,
schema changes, and retention-based deletion as separate operations.
## Data mutation support by table type
| Table type | UPDATE | DELETE | TRUNCATE |
|------------|--------|--------|----------|
| TAG | DATA correction in Standard: requires tag selection and BASETIME predicates | `BEFORE`, tag/axis predicates, or all rows | No |
| LOG | No | `BEFORE`, `OLDEST`, `EXCEPT`, or all rows | Yes |
| TRANSACTION | Yes | Yes | Yes |
| VOLATILE | Primary-key predicate | Primary-key predicate or all rows | No |
| LOOKUP | General predicates; cannot change the PK | General predicates or all rows | No |
LOG tables are designed to preserve ingested events without updating them. Store frequently
modified state or configuration in VOLATILE, LOOKUP, or TRANSACTION tables.
## Mutation units and failure handling
| Operation | Design decision |
|---|---|
| Update current settings | Key, allowed values, and concurrent-writer handling |
| Correct inaccurate measurements | Tag/time range, reason, and ROLLUP recalculation |
| Correct a LOG event | Whether to retain the original and link a correction event |
| Replace a reference key | Reference migration order and intermediate failure handling |
| Delete a period | Retention cutoff, deletion scope, and required backups |
Multiple TRANSACTION DML statements can share an explicit transaction. Do not assume LOOKUP
or VOLATILE changes or LOG or TAG ingestion participate in that transaction. For example,
TAG history may remain after a subsequent VOLATILE cache update fails. Prepare cache rebuilding
or retry procedures. Append transaction participation depends on the API and target type;
check [SDK Support Scope](/dbms/development-tools-integration/sdk-support-scope/).
Before changing multiple rows, query the count and representative rows with the same
predicates. This precheck does not lock rows or fix the later mutation scope. With concurrent
ingestion or updates, also control the operation window and target range. Verify the affected
row count and resulting values afterward.
## UPDATE policy
### TRANSACTION, VOLATILE, LOOKUP
- TRANSACTION supports general relational `UPDATE` and transactions.
- VOLATILE selects targets through primary-key equality predicates.
- LOOKUP supports general predicates, but its primary-key column cannot be changed.
LOOKUP UPDATE requires a WHERE clause. Changing a LOOKUP or VOLATILE primary key requires
separate delete and insert statements. First define how to preserve the original and switch
references. If both statements must be rolled back together, consider TRANSACTION.
For supported LOOKUP predicates and expressions, see
[LOOKUP Predicate UPDATE](/dbms/reference/sql/syntax/dml-syntax/lookup-predicate-update-syntax/).
### TAG data UPDATE
TAG time-series data and metadata use different update syntax.
| Target | Syntax | Key restriction |
|------|------|-----------|
| Time-series data | `UPDATE tag_table SET ... WHERE ...` | Requires both tag selection and BASETIME predicates |
| Metadata | `UPDATE tag_table METADATA SET ...` | Uses TAG-specific metadata syntax |
TAG data UPDATE is supported on logical TAG tables in Standard Edition. Tag names, BASETIME,
and metadata columns cannot be SET targets. If materialized rollups cover the changed range,
regenerate them with `ROLLUP_REBUILD`.
The right-hand side of TAG DATA `SET` cannot reference existing row columns. Do not apply
corrections with expressions such as `SET value = value + 1`. Pass calculated values as
constants or parameters and limit the affected range. If correction history is required,
store before/after values and reasons separately rather than only overwriting the current value.
The following references define syntax and allowed expressions:
- [TAG Data UPDATE](/dbms/reference/sql/syntax/dml-syntax/tag-data-update-syntax/)
- [TAG Data UPDATE WHERE/SET Constraints](/dbms/reference/sql/syntax/dml-syntax/tag-data-update-where-set-constraints/)
- [TAG Metadata](/dbms/tag-table-usage/tag-metadata/)
LOG tables do not support `UPDATE`.
## DELETE policy
### TRANSACTION, VOLATILE, LOOKUP
- TRANSACTION supports deletion with general `WHERE` predicates.
- VOLATILE supports primary-key deletion or deletion of all rows without a predicate.
- LOOKUP supports general predicates or deletion of all rows without `WHERE`.
For supported LOOKUP predicates, see
[LOOKUP Predicate DELETE](/dbms/reference/sql/syntax/dml-syntax/lookup-predicate-delete-syntax/).
### LOG
LOG uses retention-oriented deletion syntax instead of arbitrary `WHERE` predicates. Choose
`OLDEST`, `EXCEPT`, `BEFORE`, or deletion of all rows according to the purpose. For exact syntax
and examples, see [LOG Data Lifecycle](/dbms/log-table-usage/operations-lifecycle/).
### TAG/KV
TAG/KV can remove old data with `BEFORE` or select targets through tag names and axis predicates.
The `BEFORE` timestamp must be earlier than the current time. For syntax, see
[TAG Data Mutation](/dbms/tag-table-usage/data-input-mutation/).
For recurring retention-based deletion, use a
[Retention Policy](/dbms/operations-configuration-recovery/policy-data-retention/).
### TAG metadata
Delete TAG metadata with `DELETE FROM table_name METADATA`. If any selected tag still has
actual data, the entire statement fails.
For detailed conditions, see [TAG Metadata](/dbms/tag-table-usage/tag-metadata/).
## TRUNCATE policy
`TRUNCATE TABLE` is supported only for LOG and TRANSACTION. It removes all rows while
retaining the schema and index definitions.
| Item | TRUNCATE | DELETE |
|------|----------|--------|
| Target | Entire table | All rows or selected rows, depending on type |
| `WHERE` | Not allowed | Allowed within the supported scope |
| TRANSACTION rollback | Possible in an explicit transaction | Possible in an explicit transaction |
Before deletion, check backups and reingestion paths. For TAG, use supported `BEFORE` or
tag/axis predicates. To clear VOLATILE or LOOKUP, use `DELETE` without predicates.
Do not apply TRANSACTION rollback guarantees to LOG deletion. Later ROLLBACK cannot undo
committed changes. Logical deletion and physical disk reclamation may occur at different
times; check storage usage and the table's reclamation state.
---
title: "4.4 Anti-Patterns"
url: https://docs.machbase.com/dbms/data-modeling-table-design/table-types-patterns-type-anti/
language: en
kind: page
---
# 4.4 Anti-Patterns
An anti-pattern is a use that conflicts with data semantics and requirements, not simply the
choice of a particular table type. Check whether each example's assumptions apply to your
workload before choosing an alternative. The same schema may suit current state but not history.
- **[Unbounded History in LOOKUP](/dbms/data-modeling-table-design/table-types-patterns-type-anti/#high-frequency-lookup)**
- **[One Table per Sensor](/dbms/data-modeling-table-design/table-types-patterns-type-anti/#per-sensor-create)**
- **[Incorrect Table Type](/dbms/data-modeling-table-design/table-types-patterns-type-anti/#table-types-selection-type-wrong)**
- **[VOLATILE as Persistent Storage](/dbms/data-modeling-table-design/table-types-patterns-type-anti/#storage-persistent-volatile)**
- **[Misusing TRANSACTION for Time Series](/dbms/data-modeling-table-design/table-types-patterns-type-anti/#time-series-storage-misuse-rdb)**
## Unbounded history in LOOKUP
### Problem
The issue is not query frequency itself, but storing continuously growing measurement history
in LOOKUP. LOOKUP keeps all rows and indexes in memory, so long-term history increases memory
pressure. It suits repeated key lookups on small reference datasets.
### Anti-pattern example
```sql
-- Incorrect design: sensor measurements in LOOKUP
CREATE LOOKUP TABLE sensor_data_wrong (
sensor_id VARCHAR(64) PRIMARY KEY,
value DOUBLE,
ts DATETIME
);
-- sensor_id is the PK, so this schema holds only one current row per sensor
-- Appending history with the same PK causes a conflict
INSERT INTO sensor_data_wrong VALUES ('TEMP-01', 25.3, NOW);
INSERT INTO sensor_data_wrong VALUES ('TEMP-01', 25.5, NOW); -- Duplicate PK error
```
### Correct pattern
Consider TAG when collection and querying focus on per-tag measurement history. LOOKUP can
assign a distinct key to each measurement, but all history still resides in memory. Add a
VOLATILE latest-value cache only when performance requires it and it is rebuildable from raw
data. If TAG latest-value queries meet the requirement, no separate cache is needed.
```sql
-- Correct design: history in TAG
CREATE TAG TABLE sensor_data (
name VARCHAR(64) PRIMARY KEY,
time DATETIME BASETIME,
value DOUBLE
);
-- Latest-value cache in VOLATILE
CREATE VOLATILE TABLE sensor_latest (
sensor_id VARCHAR(64) PRIMARY KEY,
value DOUBLE,
updated_at DATETIME
);
```
### Result
| | Anti-pattern (LOOKUP) | Correct design (TAG) |
|-|-----------------|-----------------|
| History under the same sensor key | No (the sample sensor key identifies a row) | Yes (multiple measurements under a tag name) |
| Continuous ingestion path | Row-identifier based | Time-series Append API available |
| Time-range queries | General predicates | Tag/time-axis queries |
## One table per sensor
### Problem
Creating a separate table for each sensor or tag increases DDL, privileges, and query targets
as sensor counts grow, raising operational costs.
### Anti-pattern example
```sql
-- Incorrect design: one table per sensor
CREATE TAG TABLE sensor_temp_01 (
name VARCHAR(32) PRIMARY KEY, time DATETIME BASETIME, value DOUBLE
);
CREATE TAG TABLE sensor_temp_02 (
name VARCHAR(32) PRIMARY KEY, time DATETIME BASETIME, value DOUBLE
);
CREATE TAG TABLE sensor_temp_03 (
name VARCHAR(32) PRIMARY KEY, time DATETIME BASETIME, value DOUBLE
);
-- ... 10,000 sensors require 10,000 tables
```
### Consequences
| Problem | Description |
|------|------|
| Management complexity | DDL management for every table |
| Query complexity | Combining tables for cross-sensor aggregates |
| Metadata growth | Increased system catalog load |
| Adding a sensor | DDL required each time |
### Correct pattern
Store all sensor data in one TAG table with sensor names as the PRIMARY KEY.
This applies to sensors sharing column structure, privileges, and retention policies.
Separate tables may be appropriate when units, schemas, access privileges, or retention
periods require independent management. Sensor count alone should not determine table separation.
```sql
-- Correct design: all temperature sensors in one table
CREATE TAG TABLE temperature_sensor (
name VARCHAR(128) PRIMARY KEY,
time DATETIME BASETIME,
value DOUBLE
);
-- Insert all sensor data into one table
INSERT INTO temperature_sensor VALUES ('TEMP-01', NOW, 23.5);
INSERT INTO temperature_sensor VALUES ('TEMP-02', NOW, 24.1);
INSERT INTO temperature_sensor VALUES ('TEMP-10000', NOW, 22.9);
```
### Benefits
- No DDL for new sensors; INSERT with a new tag name is sufficient.
- Simpler cross-sensor aggregation.
- Fewer objects to operate and manage.
## Incorrect table type
The following common anti-patterns select table types that do not match data characteristics.
### Anti-pattern 1: Event logs in TAG
```sql
-- Incorrect: event logs in TAG
CREATE TAG TABLE error_log_wrong (
name VARCHAR(256) PRIMARY KEY, -- Event content becomes the tag name
time DATETIME BASETIME,
level SHORT
);
-- Problem: a unique name per event causes unbounded tag growth
```
**Correct design:** Use a LOG table.
```sql
CREATE LOG TABLE error_log (
level SHORT,
msg VARCHAR(512),
src VARCHAR(128)
);
```
### Anti-pattern 2: Sensor values in LOG
Storing sensor values in LOG is not inherently wrong. The problem below is omitting actual
measurement time when queries require per-tag measurement-time aggregates, leaving only
arrival time. LOG can suit searches on multi-field equipment events.
```sql
-- Insufficient schema when measurement time is required
CREATE LOG TABLE sensor_wrong (
sensor_id VARCHAR(64),
value DOUBLE
-- No measurement time; only server arrival time is recorded automatically
);
```
**Correct design:** Use a TAG table.
```sql
CREATE TAG TABLE sensor_measurements (
name VARCHAR(64) PRIMARY KEY,
time DATETIME BASETIME,
value DOUBLE
);
```
### Anti-pattern 3: Large histories in LOOKUP
```sql
-- Incorrect: order history requiring relational transactions in LOOKUP
CREATE LOOKUP TABLE order_history_wrong (
order_id LONG PRIMARY KEY,
customer VARCHAR(64)
-- Check memory for all rows/indexes and explicit transaction requirements
);
```
**Correct design:** Use a TRANSACTION table.
```sql
CREATE TRANSACTION TABLE order_history (
order_id LONG,
customer VARCHAR(64),
item_id INTEGER,
amount DOUBLE,
status VARCHAR(16)
);
-- UPDATE/DELETE/SELECT are all supported
UPDATE order_history SET status = 'SHIPPED' WHERE order_id = 1001;
```
### Anti-pattern 4: Time-series data in TRANSACTION
TRANSACTION supports time columns and an Append API, but not TAG-specific time-axis storage
or ROLLUP. Consider TAG when measurement collection and aggregation matter more than relational
changes. See
[Misusing TRANSACTION for Time Series](/dbms/data-modeling-table-design/table-types-patterns-type-anti/#time-series-storage-misuse-rdb).
## VOLATILE as persistent storage
### Problem
This pattern stores data requiring permanent retention in VOLATILE.
### Anti-pattern example
```sql
-- Incorrect: critical configuration in VOLATILE
CREATE VOLATILE TABLE critical_config (
key_name VARCHAR(64) PRIMARY KEY,
value VARCHAR(256)
);
INSERT INTO critical_config VALUES ('license_key', 'XXXX-XXXX-XXXX');
INSERT INTO critical_config VALUES ('max_connections', '1000');
-- All settings disappear on server restart!
```
### Consequences
- The table and data disappear when the server stops or restarts.
- A failure that terminates the server process also prevents recovery of in-memory data.
Keeping the only source in VOLATILE therefore risks data loss.
### Correct pattern
Store data requiring permanent retention in LOOKUP or TRANSACTION.
```sql
-- Correct: configuration in LOOKUP
CREATE LOOKUP TABLE app_config (
key_name VARCHAR(64) PRIMARY KEY,
value VARCHAR(256)
);
INSERT INTO app_config VALUES ('max_connections', '1000');
-- Data remains after server restart
```
### Appropriate VOLATILE uses
Use VOLATILE for data that can be recreated or discarded after restart. This includes task
state with a defined lifetime as well as query caches. Do not keep required business results
only in memory.
| Suitable | Unsuitable |
|------|--------|
| Latest sensor value cache | Source transaction data |
| Real-time aggregate results | Critical configuration |
| Temporary session state | Audit logs |
| Dashboard cache | User information |
## Misusing TRANSACTION for time series
### Problem
This pattern stores continuous time-series data, such as sensor or IoT measurements, in
TRANSACTION. If relational updates are unnecessary, the lack of TAG/time-axis access and
ROLLUP may conflict with query and operational requirements. Conversely, TRANSACTION is
justified when measurement registration must share a transaction with other business changes.
### Anti-pattern example
```sql
-- Incorrect: sensor time series in TRANSACTION
CREATE TRANSACTION TABLE sensor_timeseries (
sensor_id VARCHAR(64),
ts DATETIME,
value DOUBLE,
unit VARCHAR(16)
);
```
### Consequences
| Problem | Description |
|------|------|
| Ingestion semantics mismatch | Relational writes for values that do not require relational transactions |
| No TAG time axis | Cannot use TAG's BASETIME-based query structure |
| Different aggregation features | Cannot use TAG-specific ROLLUP |
### Correct pattern
Store sensor measurements in a TAG table.
```sql
-- Correct: use TAG
CREATE TAG TABLE sensor_history (
name VARCHAR(64) PRIMARY KEY,
time DATETIME BASETIME,
value DOUBLE,
unit VARCHAR(16)
);
-- Bulk ingestion through high-speed Append API buffers
-- Time-based aggregation and TAG-specific optimizations available
SELECT name, DATE_TRUNC('hour', time, 1) AS hour, AVG(value), MAX(value)
FROM sensor_history
WHERE time >= NOW - 86400000000000
GROUP BY name, hour;
```
### When TRANSACTION is appropriate
Use TRANSACTION for relational business data such as orders, inventory, and equipment
history. Even with time columns, business history requiring UPDATE/DELETE suits TRANSACTION;
high-frequency measurements that accumulate without updates suit TAG.
## Aggregating values with different meanings
Identical schemas do not make values with different units or row semantics directly
aggregatable. Average cumulative energy (kWh) is not power consumption (kW), and an unweighted
average of interval averages may differ from the overall sample average. Replacing NULL
with zero turns measurement failures into valid zeros.
Record units, sample counts, and quality rules alongside type selection. When reaggregating
interval statistics, retain required components such as sums and valid counts. Before
deleting raw data, check the resolution needed for future analysis.
---
title: "4.5 Modeling Patterns"
url: https://docs.machbase.com/dbms/data-modeling-table-design/patterns-modeling/
language: en
kind: page
---
# 4.5 Modeling Patterns
This page covers data modeling patterns commonly used in production.
Each section's SQL demonstrates a different model. Choose the relevant section and run it
in a separate practice environment after checking for existing tables with the same names.
Creating schemas does not automatically run collection, aggregation, or cache updates.
Also design the work performed by ingestion applications or schedulers and its failure handling.
- **[Time-Axis Modeling](/dbms/data-modeling-table-design/patterns-modeling/#time-axis-modeling)**
- **[Distance-Axis Modeling](/dbms/data-modeling-table-design/patterns-modeling/#distance-axis-modeling)**
- **[State and Cache Modeling](/dbms/data-modeling-table-design/patterns-modeling/#state-cache-status-modeling)**
- **[Event and Log Modeling](/dbms/data-modeling-table-design/patterns-modeling/#event-log-modeling-logs)**
- **[Reference and Master Data](/dbms/data-modeling-table-design/patterns-modeling/#reference-master-modeling)**
- **[Persistent and Temporary Data](/dbms/data-modeling-table-design/patterns-modeling/#persistent-temporary)**
- **[INSERT and UPDATE Patterns](/dbms/data-modeling-table-design/patterns-modeling/#insert-update)**
- **[JOIN and Metadata Design](/dbms/data-modeling-table-design/patterns-modeling/#join-metadata-design)**
- **[Combined Table Types](/dbms/data-modeling-table-design/patterns-modeling/#table-types-patterns-combined-type)**
## Time-axis modeling
This pattern uses time as the primary axis and suits sensor measurements, energy monitoring,
and environmental data.
One row represents one observation from one meter. Define `time` as measurement time, not
arrival time, and document whether `kwh` is a cumulative reading or interval consumption.
The following schema assumes voltage and current belong to the same observation. If their
measurement intervals or timestamps differ, store separate series or define missing-value
rules instead of forcing them into one row.
### Basic pattern: TAG table
```sql
CREATE TAG TABLE power_meter (
meter_id VARCHAR(32) PRIMARY KEY,
time DATETIME BASETIME,
kwh DOUBLE,
voltage DOUBLE,
current DOUBLE
) METADATA (
location VARCHAR(64),
phase SHORT,
rating_kw DOUBLE
);
```
### Time-range aggregation
```sql
-- Hourly average and maximum meter readings (last 24 hours)
SELECT meter_id,
DATE_TRUNC('hour', time, 1) AS hour,
AVG(kwh) AS avg_kwh,
MAX(kwh) AS peak_kwh
FROM power_meter
WHERE time >= NOW - 86400000000000
GROUP BY meter_id, hour
ORDER BY meter_id, hour;
```
If `kwh` is cumulative energy, this average is the mean meter reading, not hourly consumption.
Calculate interval consumption from the difference between start and end readings, accounting
for meter resets, replacement, and rollover. Also distinguish power (kW) from energy (kWh)
in column names and units.
### Multiple-resolution storage
Store high-resolution raw data and lower-resolution aggregates in separate tables.
```sql
-- Raw data (second-level resolution)
CREATE TAG TABLE power_raw (
meter_id VARCHAR(32) PRIMARY KEY,
time DATETIME BASETIME,
kwh DOUBLE
);
-- One-minute aggregates (cached in VOLATILE or a separate TAG table)
CREATE VOLATILE TABLE power_1min (
key_id VARCHAR(80) PRIMARY KEY,
meter_id VARCHAR(32),
ts DATETIME,
avg_kwh DOUBLE,
max_kwh DOUBLE
);
```
This DDL creates storage only. The application must assign `key_id` values in `power_1min` that
uniquely identify each meter and interval, then populate aggregates. VOLATILE results disappear
on restart, so do not use it for long-term aggregate retention. Keep statistics needed after
raw-data deletion in persistent TAG tables or supported ROLLUPs, and check each retention policy.
### Time zone handling
DATETIME represents an instant; connection time zones affect string input and output. To
display Korean time, configure the client's time zone. Adding nine hours to a stored timestamp
changes the instant itself rather than displaying the same instant in another time zone.
```sql
-- Check the client time zone, then query the original timestamp
SELECT meter_id,
time,
kwh
FROM power_meter
WHERE meter_id = 'MTR-001'
AND time >= '2024-01-01 00:00:00';
```
If input and connection time zones differ, align them before ingestion. For a JDBC `TIMEZONE`
example, see [JDBC Connections](/dbms/development-tools-integration/jdbc/).
## Distance-axis modeling
This pattern uses distance or position as the primary axis and suits pipeline inspection,
road sensors, and laser scans.
A distance axis is not another display format for time. Time-axis ROLLUP and retention rules
cannot be applied unchanged. For repeated inspections of a pipe, define an inspection-run
identifier or table separation rule so `pipe_id` does not mix runs. This example assumes
one inspection of one pipe, with distance in m and thickness in mm.
### Basic pattern
```sql
CREATE TAG TABLE pipeline_thickness (
pipe_id VARCHAR(32) PRIMARY KEY,
distance DOUBLE BASEDISTANCE, -- Unit: meters
thickness DOUBLE,
temp DOUBLE
);
```
### Query a range
```sql
-- Query the 0–50 m range for a pipe
SELECT pipe_id, distance, thickness
FROM pipeline_thickness
WHERE pipe_id = 'PIPE-A'
AND distance BETWEEN 0.0 AND 50.0
ORDER BY distance;
-- Query locations below the threshold
SELECT pipe_id, distance, thickness
FROM pipeline_thickness
WHERE pipe_id = 'PIPE-A'
AND thickness < 8.0 -- Thickness below 8 mm
ORDER BY distance;
```
### Distance-based aggregation
```sql
-- Average thickness for each 10 m segment
SELECT pipe_id,
FLOOR(distance / 10.0) * 10 AS segment_start,
AVG(thickness) AS avg_thickness,
MIN(thickness) AS min_thickness
FROM pipeline_thickness
WHERE pipe_id = 'PIPE-A'
GROUP BY pipe_id, FLOOR(distance / 10.0) * 10
ORDER BY segment_start;
```
### Combine time and distance
To manage inspection time and position together, add a distance column to a time-axis TAG table.
```sql
-- Store a time axis with position information
CREATE TAG TABLE inspection_data (
inspector VARCHAR(64) PRIMARY KEY,
time DATETIME BASETIME,
distance DOUBLE, -- Position column, not an axis
thickness DOUBLE,
defect SHORT
);
-- Query a specific date and distance range
SELECT inspector, time, distance, thickness
FROM inspection_data
WHERE inspector = 'INSPECTOR-01'
AND time BETWEEN '2024-01-01' AND '2024-01-02'
AND distance BETWEEN 100.0 AND 200.0
ORDER BY time;
```
## State and cache modeling
This cache pattern supports real-time queries of current device or sensor state.
### Latest-state cache
Cache each device's current state in a VOLATILE table.
```sql
-- State cache (VOLATILE)
CREATE VOLATILE TABLE device_status (
device_id VARCHAR(64) PRIMARY KEY,
status VARCHAR(16),
value DOUBLE,
updated_at DATETIME
);
-- State history (TAG or LOG)
CREATE TAG TABLE device_status_history (
name VARCHAR(64) PRIMARY KEY,
time DATETIME BASETIME,
value DOUBLE,
status VARCHAR(16)
);
```
### State update flow
The following SQL updates history and cache separately. The two writes are not one
transaction, and their separately evaluated `NOW` values need not match. In actual ingestion,
choose the measurement timestamp once and pass it to both paths. Define event ordering and
concurrent update handling in the application so late historical values do not overwrite
the latest cache.
```sql
-- On receiving a new measurement:
-- 1. Store history in TAG
INSERT INTO device_status_history VALUES ('DEV-01', NOW, 78.5, 'WARNING');
-- 2. Update the VOLATILE cache (ON DUPLICATE KEY UPDATE)
INSERT INTO device_status VALUES ('DEV-01', 'WARNING', 78.5, NOW)
ON DUPLICATE KEY UPDATE SET status = 'WARNING', value = 78.5, updated_at = NOW;
```
### Dashboard queries
```sql
-- All devices currently in an alarm state
SELECT device_id, status, value, updated_at
FROM device_status
WHERE status IN ('ALARM', 'WARNING')
ORDER BY updated_at DESC;
-- Latest state of a specific device
SELECT device_id, status, value, updated_at
FROM device_status
WHERE device_id = 'DEV-01';
```
### State definition reference
Manage the meaning of state codes in LOOKUP.
```sql
CREATE LOOKUP TABLE status_definition (
code VARCHAR(16) PRIMARY KEY,
label VARCHAR(64),
color VARCHAR(16),
severity SHORT
);
INSERT INTO status_definition VALUES ('NORMAL', 'Normal', 'green', 0);
INSERT INTO status_definition VALUES ('WARNING', 'Warning', 'yellow', 1);
INSERT INTO status_definition VALUES ('ALARM', 'Alarm', 'red', 2);
-- Query with JOIN
SELECT d.device_id, s.label, s.color, d.value
FROM device_status d
JOIN status_definition s ON d.status = s.code
ORDER BY s.severity DESC;
```
## Event and log modeling
Use LOG tables to model system events, alarms, and audit logs.
### Hierarchical event model
```sql
-- Alarm event LOG table
CREATE LOG TABLE alarm_event (
severity SHORT, -- 1=INFO, 2=WARN, 3=ERROR, 4=CRITICAL
category VARCHAR(32), -- Category
source VARCHAR(64), -- Event source
message VARCHAR(512),
src_ip IPV4 -- Source IP, if available
);
-- System audit LOG table
CREATE LOG TABLE audit_log (
user_id VARCHAR(64),
action VARCHAR(32), -- INSERT, UPDATE, DELETE, LOGIN, etc.
target VARCHAR(128), -- Target table/resource
detail TEXT, -- Details for full-text search
result VARCHAR(8) -- SUCCESS, FAILURE
);
CREATE INDEX idx_audit_detail ON audit_log(detail) INDEX_TYPE KEYWORD;
```
### Alarm aggregation
```sql
-- Alarm counts by severity over the last hour
SELECT severity, COUNT(*) AS cnt
FROM alarm_event
WHERE _arrival_time >= NOW - 3600000000000
GROUP BY severity
ORDER BY severity DESC;
-- Alarm summary by source over the last 24 hours
SELECT source, COUNT(*) AS total,
SUM(CASE WHEN severity = 4 THEN 1 ELSE 0 END) AS critical_cnt
FROM alarm_event
WHERE _arrival_time >= NOW - 86400000000000
GROUP BY source
ORDER BY total DESC;
```
### Log level filtering
```sql
-- ERROR or higher over the last 10 minutes
SELECT _arrival_time, source, message
FROM alarm_event
WHERE severity >= 3
AND _arrival_time >= NOW - 600000000000
ORDER BY _arrival_time DESC
LIMIT 100;
```
### Full-text search
```sql
-- Audit logs containing a keyword
SELECT _arrival_time, user_id, action, target
FROM audit_log
WHERE detail SEARCH 'password'
AND _arrival_time >= NOW - 86400000000000;
```
## Reference and master data modeling
Use LOOKUP for reference data such as code tables, equipment master data, and user information.
Storing only identifiers in history avoids repeating names and locations. However, changing
a location in current master data also changes joined historical results. If the production
line at event time matters, model separate change history with validity periods or record
those attributes in the original row. The following hierarchy does not automatically enforce
references through foreign keys; applications must reject nonexistent factory and line codes.
### Hierarchical code system
```sql
-- Main category codes
CREATE LOOKUP TABLE category_main (
code VARCHAR(8) PRIMARY KEY,
label VARCHAR(64)
);
-- Subcategory codes (reference main category)
CREATE LOOKUP TABLE category_sub (
code VARCHAR(16) PRIMARY KEY,
main_code VARCHAR(8),
label VARCHAR(64)
);
CREATE INDEX idx_sub_main ON category_sub(main_code);
```
### Equipment master hierarchy
```sql
-- Factory master
CREATE LOOKUP TABLE factory (
factory_id VARCHAR(16) PRIMARY KEY,
name VARCHAR(64),
location VARCHAR(128)
);
-- Production line master (references factory)
CREATE LOOKUP TABLE production_line (
line_id VARCHAR(16) PRIMARY KEY,
factory_id VARCHAR(16),
name VARCHAR(64)
);
-- Equipment master (references production line)
CREATE LOOKUP TABLE equipment (
equip_id VARCHAR(32) PRIMARY KEY,
line_id VARCHAR(16),
equip_name VARCHAR(128),
equip_type VARCHAR(32),
install_dt DATETIME
);
CREATE INDEX idx_equip_line ON equipment(line_id);
CREATE INDEX idx_equip_type ON equipment(equip_type);
```
### Master data joins
Store equipment identifiers in the measurement table and attributes such as factory, line,
and equipment names once in LOOKUP tables. First restrict the measurement time range, then
join LOOKUP tables by equipment identifier. For join syntax and plan inspection, see
[JOIN and Subqueries](/dbms/tag-table-usage/query-analysis/).
## Combine persistent and temporary data
Keep authoritative data in persistent tables (TAG, LOG, TRANSACTION, LOOKUP) and query caches
in VOLATILE. Do not assume the two writes share one transaction. Design for cache lag and
reconstruction after failure.
### Raw data and aggregate cache
Store raw data in a persistent table and aggregate results in VOLATILE.
```sql
-- Raw data (persistent TAG)
CREATE TAG TABLE sensor_data (
name VARCHAR(64) PRIMARY KEY,
time DATETIME BASETIME,
value DOUBLE
);
-- Aggregate cache (temporary VOLATILE)
CREATE VOLATILE TABLE sensor_recent_avg (
key_id VARCHAR(64) PRIMARY KEY,
sensor_id VARCHAR(64),
base_ts DATETIME,
avg_val DOUBLE,
max_val DOUBLE,
cnt LONG
);
```
### Cache refresh
One cache row holds a sensor's last two hours of statistics. `base_ts` is the latest included
measurement timestamp, not the aggregation window boundary. Use the same range definition
each time. For results at an identical instant, choose one reference timestamp per execution.
The example excludes future measurements. When comparing results, replace each SQL's `NOW`
with the same fixed reference timestamp to calculate both lower and upper bounds.
```sql
-- Periodic aggregate refresh (run hourly)
DELETE FROM sensor_recent_avg;
INSERT INTO sensor_recent_avg
SELECT name AS key_id,
name,
MAX(time) AS base_ts,
AVG(value),
MAX(value),
COUNT(*)
FROM sensor_data
WHERE time >= NOW - 3600000000000 * 2 -- Recalculate the last two hours
AND time <= NOW
GROUP BY name;
```
Do not append `ON DUPLICATE KEY UPDATE` to `INSERT ... SELECT`. Rebuild the cache by
deleting its contents and loading it again.
Between deletion and reload, other queries may see an empty or partially populated cache.
Define application behavior during refresh, retries after failure, and fallback to raw-data
queries. If a consistently complete result is required, consider a separate switchover or
transaction model that provides it.
### Optimize dashboard queries
```sql
-- Query the stored cache (the application handles fallback to raw data)
SELECT sensor_id, base_ts, avg_val, max_val
FROM sensor_recent_avg
WHERE base_ts >= NOW - 3600000000000 * 2
AND base_ts <= NOW
ORDER BY sensor_id, base_ts;
```
This predicate displays cache rows whose latest measurement is within the last two hours.
It does not recalculate stored averages relative to query time. Manage aggregate freshness
through the refresh interval.
### Failure recovery
If a server restart removes the VOLATILE table and cache, first recreate the table, then
recalculate the same last-two-hour statistics from TAG. Run the following CREATE only after
a restart in which the table disappeared. The source retention period must still include
the data needed for recalculation.
```sql
-- Rebuild the cache after server restart
CREATE VOLATILE TABLE sensor_recent_avg (
key_id VARCHAR(64) PRIMARY KEY,
sensor_id VARCHAR(64),
base_ts DATETIME,
avg_val DOUBLE,
max_val DOUBLE,
cnt LONG
);
INSERT INTO sensor_recent_avg
SELECT name,
name, MAX(time), AVG(value), MAX(value), COUNT(*)
FROM sensor_data
WHERE time >= NOW - 3600000000000 * 2 -- Same last-two-hour range as normal refresh
AND time <= NOW
GROUP BY name;
```
After recovery, compare per-sensor counts, averages, and latest timestamps with raw-data
aggregates at the same reference time. `cnt` includes rows with NULL measurements. Store
`COUNT(value)` separately if the sample count used for the average is needed.
## INSERT and UPDATE patterns
The model must decide which data can change. This page does not redefine the DML support
matrix. Use [Data Mutation Policy](../alter-data-mutation-policy/) for table-specific conditions
and [Data Input and Export](/dbms/development-tools-integration/data-input-load-export/) to
choose INSERT, Append, or file ingestion.
## JOIN and metadata design
Apply these principles when combining table types.
First define join cardinality. Check whether each sensor code has exactly one master row or
is reused across factories. If one source row matches several reference rows, result counts
and aggregates such as SUM can be duplicated. Match identity scope and types, not just code names.
1. Review join order using filtered row counts and execution plans, not only raw row counts.
2. Match JOIN column types and check supported index usage.
3. Reduce joined rows with explicit time ranges and business predicates in WHERE.
4. For queries that include TAG attributes, consider METADATA instead of separate LOOKUP tables.
For reproducible join examples, see [TAG Queries and Analysis](/dbms/tag-table-usage/query-analysis/)
and [LOOKUP Queries and Analysis](/dbms/lookup-table-usage/query-analysis/).
## Combined table types
These common designs combine multiple table types in production systems.
### Manufacturing equipment monitoring
```text
┌───────────────────────────────────────────────────────────────┐
│ Equipment monitoring system │
├───────────────────┬───────────────────┬───────────────────────┤
│ TAG │ LOG │ LOOKUP │
│ sensor_data │ alarm_event │ equipment_master │
│ Measurement │ Alarm events │ Equipment reference │
│ history │ │ data │
├───────────────────┴───────────────────┴───────────────────────┤
│ VOLATILE │
│ sensor_latest (latest-value cache) │
└───────────────────────────────────────────────────────────────┘
```
### Logistics and order management (Standard Edition)
```text
┌───────────────────────────────────────────────────────────────┐
│ Logistics management system │
├───────────────────┬───────────────────┬───────────────────────┤
│ TRANSACTION │ LOG │ LOOKUP │
│ orders │ delivery_log │ product_master │
│ Order management │ Delivery events │ Product reference │
│ UPDATE/DELETE │ │ data │
├───────────────────┴───────────────────┴───────────────────────┤
│ VOLATILE │
│ order_status_cache (current-state cache) │
└───────────────────────────────────────────────────────────────┘
```
### Pattern summary
| Role | Recommended type | Reason |
|------|---------|------|
| High-frequency measurement history | TAG | High-speed Append API buffers, time-series optimization |
| Event and alarm logs | LOG | Append-only, automatic arrival time |
| Relational business data (UPDATE/DELETE) | TRANSACTION | SELECT/INSERT/UPDATE/DELETE all supported |
| Reference and code data | LOOKUP | PK identity, general-predicate UPDATE/DELETE, persistence |
| Real-time state cache | VOLATILE | In-memory speed, UPSERT |
---
Read next:
- [SELECT GROUP BY and Aggregation](/dbms/reference/sql/syntax/select-syntax/)
- [Operations and Configuration](/dbms/operations-configuration-recovery/)
---
title: "5. Using TAG Tables"
url: https://docs.machbase.com/dbms/tag-table-usage/
language: en
kind: section
---
# 5. Using TAG Tables
TAG tables store measurement histories by repeatedly observed entity name
and a time or distance axis. This chapter covers Machbase DBMS 8.7.0
TAG structure, ingestion, queries, metadata, correction, and operations.
It validates the deployment and data-model choices from Chapters 3 and 4
through practical SQL.
A tag is an observed entity; a DATA row is one observation. METADATA holds
one attribute row per tag, not separate attributes per historical
observation. Distinguish original measurements, current attributes,
interval aggregates, and ingestion times to interpret results and
correction scope consistently.
## Chapter Contents
| Section | Content |
|---|---|
| [Overview and Selection Criteria](./overview-use-criteria/) | Tag identifiers, observation rows, time/distance axes |
| [Table Structure and Schema](./table-structure-schema/) | Column order/types, LSL/USL, BINARY, storage design |
| [Create, Alter, and Drop](./create-alter-drop/) | Basic DDL, METADATA expansion, object cleanup |
| [Data Ingestion and Changes](./data-input-mutation/) | Automatic registration, SQL/Append/file input |
| [Queries and Analysis](./query-analysis/) | Range/latest-value/STAT queries and interpretation |
| [Indexes and Performance](./index-performance/) | Validate access paths with actual data |
| [Operations and Data Lifecycle](./operations-lifecycle/) | Deletion, retention, deduplication completion |
| [Constraints, Errors, and Troubleshooting](./constraints-errors-troubleshooting/) | Valid conditions and intentional failure examples |
| [Usage Patterns and Scenarios](./patterns-scenarios/) | Observation granularity, units, missing-data models |
| [TAG Metadata](./tag-metadata/) | Registration, queries, changes, deletion, JSON, ARRAY |
| [TAG Data UPDATE and Correction](./tag-data-update-correction/) | Direct correction, NULL correction, audit history |
| [tagmetaimport and Bulk Metadata Registration](./tagmetaimport/) | CSV setup, input targets, rerun errors |
## Exercises and Support Scope
Each page is an independent exercise. If a setup table exists, determine
whether it belongs to another workload; do not delete it arbitrarily.
Run success examples separately from intentional failures. Apply cleanup
SQL only to objects created for that exercise.
TAG DATA UPDATE is Standard Edition only. Check Edition-specific scope
for duplicate-check intervals, METADATA ALTER, and individual LSL/USL
operations. SQL INSERT, Append responses, storage-buffer flush, and
index/statistics completion are different events.
[Chapter 6](../tag-rollup-usage/) covers full ROLLUP creation, query, and
rebuild procedures. This chapter addresses only how TAG corrections and
deletions relate to aggregates.
---
title: "5.1 Overview and Selection Criteria"
url: https://docs.machbase.com/dbms/tag-table-usage/overview-use-criteria/
language: en
kind: page
---
# 5.1 Overview and Selection Criteria
Consider TAG for histories of repeated observations of sensors, equipment,
or similar entities. Start by distinguishing one tag from one row. Multiple
rows at different times can share a name; a shared name alone does not
remove duplicate rows.
## TAG Table Characteristics
`PRIMARY KEY` identifies the tag name, unlike a relational per-row unique
key. One table has either a time axis or a distance axis.
```sql
-- Time-axis TAG for observations over time.
CREATE TAG TABLE ch5_overview_time (
name VARCHAR(64) PRIMARY KEY,
time DATETIME BASETIME,
value DOUBLE SUMMARIZED
);
-- Distance-axis TAG for observations over distance or position.
-- Distance-axis TAG does not support ROLLUP.
CREATE TAG TABLE ch5_overview_distance (
name VARCHAR(64) PRIMARY KEY,
distance DOUBLE BASEDISTANCE,
value DOUBLE
);
```
In both examples, the name is first and the axis second. Time axes use
`DATETIME BASETIME`; distance axes use `DOUBLE`, `LONG`, or `ULONG` with
`BASEDISTANCE`. Optional `SUMMARIZED` belongs on the third column. See
[Create, Alter, and Drop](../create-alter-drop/) for column order/types and
[Table Structure and Schema](../table-structure-schema/) for design choices.
`SUMMARIZED` marks a representative statistics/aggregation column. It does
not automatically create a ROLLUP object.
The following lists data scopes to distinguish, not columns of the sample
table. DATA consists of user-supplied observations; METADATA and STAT are
separate scopes for per-tag attributes and statistics.
| Scope | Meaning |
|---|---|
| Tag name | First column identifying a sensor or repeatedly observed entity |
| DATA | User-supplied observation rows: axis values and DATA columns |
| METADATA | Current per-tag attributes such as location, units, and settings; separate from ordinary DATA |
| STAT | Per-tag ingestion statistics from `V$
_STAT`, not direct sample-table columns |
## When TAG Fits
- Continuously append histories of multiple entities using one schema.
- Frequently query time/distance ranges for specific tags.
- Select tags by current attributes and analyze their histories.
- Store and query repeated interval statistics on a time axis. See
[TAG ROLLUP](../../tag-rollup-usage/) for aggregation design.
A single-value model uses a tag per measurement; a multi-value model groups
temperature, pressure, and other simultaneous measurements in one row. Do
not force readings from different times into one row; define missing-data
and quality policies. Equal timestamps may reflect duplicate collection,
so define duplicate handling for name, time, and values separately.
## When to Consider Other Tables
| Requirement | Alternative |
|---|---|
| Event searches centered on events rather than observed entities | LOG |
| General predicates and changes on persistent reference data | LOOKUP |
| Explicit multi-DML transactions and relational changes | TRANSACTION (Standard Edition only) |
| Shared state cache rebuildable from a source | VOLATILE |
TAG also supports JOIN and limited value correction. Do not assume that JOIN
requirements rule out TAG or that every measurement must use TAG.
DATA UPDATE is Standard Edition only. Cluster supports only METADATA UPDATE;
plan reingestion and reaggregation when measurement correction is required.
## Design Sequence
1. Define whether a tag identifies a sensor, equipment item, or inspection run.
2. Define the meaning and units of measurement time or distance/position.
3. Define value types, NULL/quality indicators, and observation frequency.
4. Separate per-observation attributes from current METADATA.
5. Compare range-query, aggregation, correction, and retention needs with Edition support.
Verify and remove the example tables as follows:
```sql
-- Inspect the example schemas.
DESC ch5_overview_time;
DESC ch5_overview_distance;
-- Clean up to avoid name conflicts with later examples.
DROP TABLE ch5_overview_distance;
DROP TABLE ch5_overview_time;
```
Continue with [Schema Design](../table-structure-schema/) and
[Ingestion and Query Exercises](../data-input-mutation/).
---
title: "5.2 Table Structure and Schema"
url: https://docs.machbase.com/dbms/tag-table-usage/table-structure-schema/
language: en
kind: page
---
# 5.2 Table Structure and Schema
## Design a TAG Table
TAG table design determines what constitutes one tag and where each value belongs,
not just how many columns to create.
See [Create, Alter, and Drop](../create-alter-drop/#original-85-creating-tag-tables)
for positional column roles, axis types, and prohibited types. This page covers
decisions within those rules.
The DDL on this page contains independent model examples. Sections explain required
creation order for exercises. Use different names if objects already exist.
Review the following when designing a schema:
- [Use Cases](#tag-schema-use-case-summary)
- [Tag Name Column](#tag-name-column-design)
- [Choose a Time or Distance Axis](#time-axis-design-tag)
- [Value Columns](#tag-table-design-design-column)
- [METADATA Columns](#metadata-column-design-summary)
- [JSON METADATA Columns](#json-metadata-column-design-summary)
- [Binary Columns](#tag-table-design-design-column-binary)
- [VARCHAR Storage Optimization](#tag-table-design-storage-varchar)
- [Storage Strategy](#tag-table-design-strategy)
- [LSL/USL](#tag-table-design-lsl-usl)
- [Correction and Duplicate Policies](#correction-duplication-policy-summary)
- [Constraints and Support](#tag-schema-limitations-summary)
### Use Cases
TAG suits repeated observations with a shared structure across many subjects. Identify
the observed entity, such as a sensor, equipment item, vehicle, or inspection run,
then check whether its history can be queried along time or distance. Decide here
whether to model it as TAG or split it into other table types. See
[Use Cases](../patterns-scenarios/#use-cases-tag) for business models.
### Tag Name Column
Decide whether a tag represents a sensor, equipment item, or inspection run. Finer
granularity increases tag counts and metadata/index overhead; coarser granularity
mixes histories of different entities within one tag. Naming conventions are also
covered in [VARCHAR Storage Optimization](#tag-table-design-storage-varchar).
### Choose a Time or Distance Axis
Choose the axis according to the query range: measurement timestamps require a time
axis; cumulative positions along a route require a distance axis. One TAG table
cannot have both. Changing the axis later requires recreating the table.
```sql
-- Time-axis TAG with ranges based on measurement timestamps.
CREATE TAG TABLE time_sensor (
name VARCHAR(40) PRIMARY KEY,
time DATETIME BASETIME,
value DOUBLE SUMMARIZED
);
```
```sql
-- Distance-axis TAG with ranges based on distance or position.
CREATE TAG TABLE rail_sensor (
name VARCHAR(40) PRIMARY KEY,
distance DOUBLE BASEDISTANCE,
value DOUBLE
);
```
`DURATION`, ROLLUP, and time functions apply only to BASETIME. Use ordinary comparisons
and `BETWEEN` for distance ranges. See [Queries and Analysis](../query-analysis/) for examples.
### Value Columns
Value columns are ordinary data columns other than the tag name and axis columns.
They store values that change per measurement row, such as readings, status, and quality codes.
#### Supported Types
Common types are listed below. See the [Data Type Reference](/dbms/reference/sql/types/)
for full support, including JSON, BINARY, DECIMAL, and numeric ARRAY.
| Type | Description | Storage size |
|------|------|---------|
| `DOUBLE` | 64-bit floating point | 8 bytes |
| `FLOAT` | 32-bit floating point | 4 bytes |
| `LONG` | 64-bit integer | 8 bytes |
| `INTEGER` (`INT`) | 32-bit integer | 4 bytes |
| `SHORT` | 16-bit integer | 2 bytes |
| `VARCHAR(n)` | Variable-length string | Up to n bytes |
#### Recommended Types
| Data | Recommended type |
|--------|---------|
| Analog values: temperature, humidity, pressure | `DOUBLE` |
| Counters, status codes | `INTEGER` |
| Flags, binary states | `SHORT` |
| Cumulative energy or flow | `DOUBLE` or `LONG` |
| Tag string values | `VARCHAR(n)` |
#### Multiple Value Columns
Storing multiple measurements in one table may introduce NULLs. This model suits
measurements collected at the same time.
```sql
CREATE TAG TABLE weather_station (
name VARCHAR(64) PRIMARY KEY,
time DATETIME BASETIME,
temperature DOUBLE, -- Always collected
humidity DOUBLE, -- Always collected
wind_speed DOUBLE, -- Optional
rainfall DOUBLE -- Optional
);
```
#### Allow NULL Values
TAG value columns allow NULL by default. If a tag collects only some measurements,
insert NULL into the remaining columns.
```sql
-- Without wind_speed and rainfall
INSERT INTO weather_station VALUES ('WS-01', NOW, 22.5, 65.0, NULL, NULL);
```
### METADATA Columns
Use METADATA for current per-tag attributes, not values repeated in every DATA row.
Examples include installation location, units, equipment settings, and management
status. Separating DATA and METADATA preserves observation history while allowing
independent queries and changes to current tag attributes. Decide which attributes
belong in METADATA and which remain DATA: values that change per observation belong
in DATA; mostly fixed attributes over a tag lifetime belong in METADATA. See
[Use METADATA](../tag-metadata/#original-85-tag-metadata) for input, query, and update examples.
### JSON METADATA Columns
Consider JSON METADATA for hierarchical or frequently changing attribute sets. For
example, store equipment location, manufacturer details, and installation options
in one JSON document and query selected paths. Choose separate columns for fixed
attributes used frequently in predicates, or JSON when attributes vary by tag.
Consider indexes for frequently filtered paths. See
[JSON METADATA](../tag-metadata/#metadata-design-json) for syntax and examples.
### Binary Columns
Use `BINARY(n)` in TAG tables for sensor frames of 1–32767 bytes. See
[Binary Columns](#original-85-binary-columns) for input literals, length limits, and
driver behavior. For large images or waveforms, also consider external storage
with only a reference key in the table.
### VARCHAR Storage Optimization
Declare `VARCHAR` according to the actual maximum length. See the
[DDL Reference](/dbms/reference/sql/syntax/ddl-syntax/) for storage-option syntax.
Combine site, equipment, and sensor identifiers with consistent separators in tag
names to support range queries.
### Storage Strategy
Data is stored in column storage separated by tag. Choose a strategy based on volume and query patterns.
#### One Table or Multiple Tables
##### One TAG Table (Recommended)
Group sensors of the same kind in one TAG table.
```sql
-- Recommended: all temperature sensors in one table
CREATE TAG TABLE temperature_sensor (
name VARCHAR(64) PRIMARY KEY,
time DATETIME BASETIME,
value DOUBLE
);
```
**Benefits**
- Fewer objects to manage
- Easier cross-tag aggregation
- Simpler operations
##### Multiple TAG Tables
Consider separate tables for different column layouts, retention periods, privileges,
or operational cycles. Do not create one table per sensor merely because sensor
counts increase.
```sql
-- Temperature/humidity sensors (DOUBLE values)
CREATE TAG TABLE thermo_sensor (
name VARCHAR(64) PRIMARY KEY,
time DATETIME BASETIME,
temp DOUBLE,
humid DOUBLE
);
-- Vibration sensors (DOUBLE + BINARY waveform)
CREATE TAG TABLE vibration_sensor (
name VARCHAR(64) PRIMARY KEY,
time DATETIME BASETIME,
rms DOUBLE,
waveform BINARY(4096)
);
```
#### Manage Tag Counts
- Measure tag-index and metadata memory growth with production-scale data.
- Encode sensor hierarchies in tag names.
- Avoid designs where every record has a unique tag name (an antipattern).
#### Partition Strategy
Limit time-axis TAG queries by `BASETIME`. Do not depend on internal storage objects
or partition names. Manage retention through
[Data Retention Policies](/dbms/operations-configuration-recovery/policy-data-retention/).
### LSL and USL
LSL (Lower Specification Limit) and USL (Upper Specification Limit) define acceptable
value bounds. Setting per-tag bounds in METADATA can reject out-of-range DATA input,
allowing different ingestion quality rules for each tag.
This feature rejects input; it does not correct values. Log and reprocess rejected
input according to the ingestion error policy.
#### Constraints
The following constraints apply:
* LSL/USL is not entirely unsupported in Cluster. Limit definitions at table creation,
metadata values, and DATA INSERT/Append limit checks use common paths. The exercise
below that adds limit columns to existing METADATA with ALTER is for Standard Edition.
* The third TAG column, __Value__, must be __SUMMARIZED__ to configure LSL/USL.
* LSL must be less than or equal to USL. Input __Value__ must be within the inclusive
bounds: __(LSL <= Value <= USL)__.
* Data inserted before limits are configured is not validated.
* NULL LSL/USL columns disable the corresponding input validation.
* Limits can be used independently; configure only USL for an upper bound.
* USL alone checks only upper-bound violations; LSL alone checks only lower-bound violations.
#### Supported Data Types
Limit columns must have the same type as the target __Value__ column. The following
covers bounds for basic numeric types. SUMMARIZED itself also supports JSON, so
SUMMARIZED eligibility and numeric limit configuration are separate requirements.
| Type | Description | Range | Significant digits |
|----|------|-----|----|
| short | Signed 16-bit integer | -32767 ~ 32767 | - |
| ushort | Unsigned 16-bit integer | 0 ~ 65534 | - |
| integer | Signed 32-bit integer | -2147483647 ~ 2147483647 | - |
| uinteger | Unsigned 32-bit integer | 0 ~ 4294967294 | - |
| long | Signed 64-bit integer | -9223372036854775807 ~ 9223372036854775807 | - |
| ulong | Unsigned 64-bit integer | 0~18446744073709551614 | - |
| float | 32-bit floating point | - | 6[^1] |
| double | 64-bit floating point | - | 15[^1] |
#### Configure and Use LSL/USL
The CREATE examples show independent alternatives. Only the base `example` table is
used in subsequent INSERT/UPDATE exercises; create alternatives separately.
Specify `LOWER LIMIT` (LSL) or `UPPER LIMIT` (USL) on tag metadata columns, either when
creating the TAG table or adding metadata columns.
##### CREATE
```sql
CREATE TAG TABLE example (
tag_id VARCHAR(50) PRIMARY KEY,
time DATETIME BASETIME,
value INTEGER SUMMARIZED)
METADATA (
lsl INTEGER LOWER LIMIT,
usl INTEGER UPPER LIMIT
);
```
Use both limit columns or only one. LSL alone checks `Value >= LSL` without an upper
bound, equivalent to a NULL USL value.
```sql
CREATE TAG TABLE example_lower_only (
tag_id VARCHAR(50) PRIMARY KEY,
time DATETIME BASETIME,
value INTEGER SUMMARIZED)
METADATA (
lsl INTEGER LOWER LIMIT
);
```
##### ADD COLUMN
When added with `ADD COLUMN` after data already exists, the default is __NULL__.
```sql
CREATE TAG TABLE example_alter_limits (
tag_id VARCHAR(50) PRIMARY KEY,
time DATETIME BASETIME,
value INTEGER SUMMARIZED
);
ALTER TABLE example_alter_limits METADATA ADD COLUMN (lsl INTEGER LOWER LIMIT);
ALTER TABLE example_alter_limits METADATA ADD COLUMN (usl INTEGER UPPER LIMIT);
```
As with [CREATE](#create), you can add only one limit attribute.
```sql
CREATE TAG TABLE example_alter_upper (
tag_id VARCHAR(50) PRIMARY KEY,
time DATETIME BASETIME,
value INTEGER SUMMARIZED
);
ALTER TABLE example_alter_upper METADATA ADD COLUMN (usl INTEGER UPPER LIMIT);
```
##### INSERT
Set LSL/USL values for a specific TAG ID.
```sql
INSERT INTO example metadata VALUES ('TAG_01', 100, 200);
```
Subsequent tag data input behaves as follows:
```sql
INSERT INTO example VALUES ('TAG_01', NOW, 95); -- Failure
```
```text
[ERR-02342: SUMMARIZED value is less than LOWER LIMIT.]
```
```sql
INSERT INTO example VALUES ('TAG_01', NOW, 100); -- Success (Inclusive)
```
```text
1 row(s) inserted.
Elapsed time: 0.000
```
```sql
INSERT INTO example VALUES ('TAG_01', NOW, 150); -- Success
```
```text
1 row(s) inserted.
Elapsed time: 0.000
```
```sql
INSERT INTO example VALUES ('TAG_01', NOW, 200); -- Success (Inclusive)
```
```text
1 row(s) inserted.
Elapsed time: 0.000
```
```sql
INSERT INTO example VALUES ('TAG_01', NOW, 205); -- Failure
```
```text
[ERR-02341: SUMMARIZED value is greater than UPPER LIMIT.]
```
Querying the TAG table shows that only values within the specification range were inserted.
```sql
SELECT * FROM example;
```
```text
TAG_ID TIME VALUE LSL USL
------------------------------------------------------------------------------------------------------------------------------
TAG_01 2023-09-12 09:31:27 923:289:631 100 100 200
TAG_01 2023-09-12 09:31:27 929:013:232 150 100 200
TAG_01 2023-09-12 09:31:27 939:209:248 200 100 200
[3] row(s) selected.
Elapsed time: 0.001
```
##### UPDATE
Update LSL/USL column values. These changes do not apply retroactively to existing data.
```sql
UPDATE example metadata SET lsl = 10, usl = 100 WHERE tag_id = 'TAG_01';
```
```text
1 row(s) updated.
Elapsed time: 0.001
```
```sql
SELECT tag_id, lsl, usl FROM example METADATA;
```
```text
TAG_ID LSL USL
----------------------------------------------------------------------------------------
TAG_01 10 100
[1] row(s) selected.
Elapsed time: 0.001
```
##### DELETE
Disable LSL/USL constraints by setting their values to NULL, not by using `DROP COLUMN`.
```sql
UPDATE EXAMPLE METADATA SET lsl = NULL, usl = NULL WHERE tag_id = 'TAG_01';
```
```text
1 row(s) updated.
Elapsed time: 0.001
```
```sql
SELECT tag_id, lsl, usl FROM example METADATA;
```
```text
TAG_ID LSL USL
----------------------------------------------------------------------------------------
TAG_01 NULL NULL
[1] row(s) selected.
Elapsed time: 0.001
```
#### Check LSL/USL Violations in TRACE Logs
- Location: `$MACHBASE_HOME/trc/machbase.trc`
- Quick filter:
```bash
grep LIMIT_DROP $MACHBASE_HOME/trc/machbase.trc | tail -n 20
```
- Log format: `LIMIT_DROP (TYPE=) TABLE= TAG=`
- TYPE=LOWER/UPPER identifies the violated bound.
- DATETIME uses `YYYY-MM-DD HH24:MI:SS mmm:uuu:nnn`.
- Actual examples:
```
[2025-11-29 13:50:34 P-151395 T-126343511537344][QP-INFO] LIMIT_DROP (TYPE=LOWER) TABLE=TAG3 TAG=tag-1 TIME=2020-01-01 00:00:00 000:000:000 VALUE=5.55
[2025-11-29 13:50:34 P-151395 T-126343511537344][QP-INFO] LIMIT_DROP (TYPE=UPPER) TABLE=TAG3 TAG=tag-1 TIME=2020-01-01 00:00:04 000:000:000 VALUE=30.55
[2025-11-29 13:50:35 P-151395 T-126344475694784][QP-INFO] LIMIT_DROP (TYPE=LOWER) TABLE=TAG3 TAG=tag-2 TIME=1998-12-24 09:00:00 000:000:000 VALUE=0
[2025-11-29 13:50:35 P-151395 T-126344475694784][QP-INFO] LIMIT_DROP (TYPE=UPPER) TABLE=TAG3 TAG=tag-2 TIME=1998-12-24 09:00:00 000:000:008 VALUE=45
```
- Usage
- Identify LOWER/UPPER violation times and values by tag.
- Filter further by TAG/table name with grep to trace specific targets.
- Caution: Lines are limited to about 4KB and may truncate with many columns. Immediately after startup, before the metadata cache is ready, table names may appear as IDs.
[^1]: [IEEE 754](https://en.wikipedia.org/wiki/IEEE_754)
### Correction and Duplicate Policies
Value correction and deduplication require decisions beyond column definitions, but
should be planned during schema design. Decide which value columns can change,
whether to preserve originals in separate columns or tables, and how to rebuild
ROLLUP after correction. DATA UPDATE is Standard Edition only; in Cluster, design
reingestion and reaggregation procedures. See
[Data Correction](../tag-data-update-correction/#design-correction-tag).
If the same tag/axis value can be inserted repeatedly, decide whether to allow
duplicates, remove them during collection, or use Machbase automatic deduplication.
See [Automatic Deduplication](../operations-lifecycle/#original-85-duplication-removal)
for configuration and operational validation.
### Constraints and Support
TAG tables are designed for repeated observations and do not support every SQL
feature of ordinary relational tables. Check axis, METADATA, correction, ROLLUP, and
Edition support before implementation. Verify that the design fits the supported
scope and revisit affected decisions if needed. See
[Constraints and Precautions](../constraints-errors-troubleshooting/#limitations-tag)
for unsupported features and common errors.
## Binary Columns
`BINARY(n)` stores fixed-length sensor-frame binary values in TAG tables. TAG
`BINARY` without a length uses 32767 bytes. Specify the required frame size to make
storage and transmission size clear. The length-qualified `BINARY(n)` form cannot
be declared in other table types. Valid lengths are 1–32K-1 (1–32767) bytes.
Binary columns cannot be indexed.
Insert `BINARY` values with explicit binary literals.
### DDL Rules
```sql
CREATE TAG TABLE t1(
name VARCHAR(32) PRIMARY KEY,
time DATETIME BASETIME,
frame BINARY(4)
);
```
- Valid length: `1 <= n <= 32767` (32K-1).
- Out-of-range lengths, such as `BINARY(0)`, fail at creation.
- `DESC` and table metadata show declared byte length, not hexadecimal width.
SQL `LENGTH(binary_col)` returns the displayed value length excluding trailing
zero padding added to short input.
### Supported Input Formats
```sql
X'hex_digits'
x'hex_digits'
B'bit_digits'
b'bit_digits'
O'octal_digits'
o'octal_digits'
```
| Format | Meaning | Unit |
| --- | --- | --- |
| `X'...'`, `x'...'` | Hexadecimal literal | 2 hexadecimal digits = 1 byte |
| `B'...'`, `b'...'` | Binary literal | 8 bits = 1 byte |
| `O'...'`, `o'...'` | Octal literal | 3 octal digits = 1 byte |
Prefixes are case-insensitive.
```sql
CREATE TAG TABLE t_bin (
name VARCHAR(20) PRIMARY KEY,
time DATETIME BASETIME,
value BINARY(4)
);
INSERT INTO t_bin VALUES('hex1', '2024-01-01 00:00:00', X'0A');
INSERT INTO t_bin VALUES('hex2', '2024-01-01 00:00:01', x'00010203');
INSERT INTO t_bin VALUES('bit1', '2024-01-01 00:00:02', B'00001010');
INSERT INTO t_bin VALUES('oct1', '2024-01-01 00:00:03', O'012');
```
`X'0A'`, `B'00001010'`, and `O'012'` all represent the one-byte value `0x0A`.
### Binary Literal Rules
#### Hexadecimal Literals
`X'...'` and `x'...'` accept `0-9`, `A-F`, and `a-f`.
```sql
X'00'
X'0AFF'
x'abcdef'
```
Hexadecimal literals require an even number of digits; two digits represent one byte.
#### Binary Literals
`B'...'` and `b'...'` accept only `0` and `1`.
```sql
B'00000000' -- 0x00
B'00001010' -- 0x0A
b'11111111' -- 0xFF
```
The bit count must be a multiple of 8; eight bits represent one byte.
#### Octal Literals
`O'...'` and `o'...'` accept only `0-7`.
```sql
O'000' -- 0x00
O'012' -- 0x0A
o'377' -- 0xFF
```
Octal digits must occur in groups of three. Each group must be in the one-byte
range `000` through `377`.
#### Empty Values
Empty single quotes represent a zero-length binary value.
```sql
X''
B''
O''
```
### Length Limits
A `BINARY(n)` column accepts at most `n` bytes.
```sql
CREATE TAG TABLE t_limit (
name VARCHAR(20) PRIMARY KEY,
time DATETIME BASETIME,
value BINARY(2)
);
INSERT INTO t_limit VALUES('ok_hex', '2024-01-01 00:00:00', X'0AFF');
INSERT INTO t_limit VALUES('ok_bit', '2024-01-01 00:00:01', B'0000101011111111');
INSERT INTO t_limit VALUES('ok_oct', '2024-01-01 00:00:02', O'012377');
INSERT INTO t_limit VALUES('bad_hex', '2024-01-01 00:00:03', X'000102'); -- Fails: 3 bytes
```
Input fails whenever the final binary value exceeds the target `BINARY(n)` length,
regardless of source. This applies to binary literals, ordinary strings, legacy
`'0x...'` string input, and copying other `BINARY` columns through `INSERT ... SELECT`.
```sql
CREATE TAG TABLE t_src (
name VARCHAR(20) PRIMARY KEY,
time DATETIME BASETIME,
value BINARY(8)
);
CREATE TAG TABLE t_dst (
name VARCHAR(20) PRIMARY KEY,
time DATETIME BASETIME,
value BINARY(4)
);
INSERT INTO t_src VALUES('k1', '2024-01-01 00:00:00', X'0102030405060708');
INSERT INTO t_dst SELECT name, time, value FROM t_src; -- Fails: 8-byte value into BINARY(4)
```
The same length check applies to values produced inside SQL expressions, including
`CASE`, `INSERT ... SELECT`, and views.
### Invalid Input
The following inputs are invalid:
```sql
X'0' -- Odd number of hexadecimal digits
X'0G' -- G is not a hexadecimal digit
B'0101' -- Bit count is not a multiple of 8
B'00000002' -- 2 is not a binary digit
O'12' -- Octal digits are not in groups of three
O'400' -- Exceeds one-byte range
X'0102 -- Missing closing single quote
```
Invalid values or excessive lengths fail with this error:
```text
[ERR-02233: Error occurred at column (n): (Invalid insert value.)]
```
### Differences from Legacy String Input
String input in the form `'0x...'` remains available for compatibility. It converts
a string to `BINARY`; `X'...'`, `B'...'`, and `O'...'` explicitly identify binary
values as SQL binary literals.
Ordinary strings can also be inserted into `BINARY(n)`, but fail if their byte
length exceeds `n`. Prefer explicit binary literals in new SQL for clarity.
`'0b...'`, `'0o...'`, and unquoted `0x...`, `0b...`, `0o...` are not supported
as binary literals.
### Output and Tools
- machsql displays uppercase hexadecimal without `0x`. Trailing zero padding added
to short input is omitted from text output.
- machloader: Declare `BINARY(n)` in the schema; invalid or oversized values fail.
- Machbase SQLCLI, ODBC, Java, C#, and Node.js drivers send/receive fixed-length
buffers; metadata `LENGTH` is in bytes.
## Clean Up Examples
Drop only tables actually created on this page. If you did not execute an
alternative DDL statement, do not run its DROP statement.
```sql
DROP TABLE time_sensor;
DROP TABLE rail_sensor;
DROP TABLE weather_station;
DROP TABLE temperature_sensor;
DROP TABLE thermo_sensor;
DROP TABLE vibration_sensor;
DROP TABLE example;
DROP TABLE example_lower_only;
DROP TABLE example_alter_limits;
DROP TABLE example_alter_upper;
DROP TABLE t1;
DROP TABLE t_bin;
DROP TABLE t_limit;
DROP TABLE t_dst;
DROP TABLE t_src;
```
---
title: "5.3 Create, Alter, and Drop"
url: https://docs.machbase.com/dbms/tag-table-usage/create-alter-drop/
language: en
kind: page
---
# 5.3 Create, Alter, and Drop
TAG tables require a tag identifier and one axis column. This page provides runnable
basic examples and links to the SQL reference for complete options.
## Create a TAG Table
The first two columns have fixed roles. The name and axis columns are required;
changing their order or placing them elsewhere causes creation to fail.
| Position | Purpose and properties |
| --- | --- |
| First | Tag name: `VARCHAR` with `PRIMARY KEY`; no other type is allowed. Identifies a repeatedly observed entity such as a sensor, equipment item, or inspection run. Multiple DATA rows can share a tag name, unlike a relational per-row unique key. |
| Second | Orders and queries observations by time or distance/position. Use `DATETIME BASETIME` for time or `DOUBLE`, `LONG`, or `ULONG` with `BASEDISTANCE` for distance. A TAG table has only one of these axes. |
| Third and later | DATA columns that vary per observation, such as temperature, pressure, status, or quality code. Support numeric types, `VARCHAR`, `DATETIME`, JSON, numeric ARRAY, and BINARY. Choose one representative value for per-tag statistics and automatic ROLLUP by marking it `SUMMARIZED`. This is optional and allowed only on the third column. |
ARRAY is allowed only in other DATA columns, not the name, axis, or `SUMMARIZED`
column. Unlike LOG, TAG DATA does not support `TEXT`, `CLOB`, or `BLOB`; use
`VARCHAR` for long strings and `BINARY` for binary data. See the
[Data Type Reference](/dbms/reference/sql/types/) for syntax and ranges.
Declare attributes stored once per tag separately in `METADATA`, outside those
column positions. `location` in the time-axis example is one such attribute.
The two tables below are independent exercises. Check for existing objects,
then execute in order. Choose an axis matching the data meaning. Distance-axis
TAG tables do not support ROLLUP.
### Time-Axis TAG
```sql
CREATE TAG TABLE ch5_tag_ddl (
name VARCHAR(32) PRIMARY KEY,
time DATETIME BASETIME,
value DOUBLE SUMMARIZED
) METADATA (
location VARCHAR(64)
);
```
`SUMMARIZED` has two effects. First, per-tag STAT views collect value statistics
such as `MIN_VALUE` and `MAX_VALUE` for this column. Without it, STAT retains
only row counts and axis ranges. Second, automatic `WITH ROLLUP` creation and
whole-JSON-document ROLLUP use this column.
Ordinary numeric ROLLUP specifies a column explicitly in
`CREATE ROLLUP ... ON table(column)`, so it does not require `SUMMARIZED`. Omit
it if value statistics are unnecessary and you plan to create ROLLUP manually.
Supported types are the supported numeric types and JSON. See
[Chapter 6](../../tag-rollup-usage/) for ROLLUP creation requirements.
Store per-tag attributes such as location and units in `METADATA`; define
values that change per measurement as ordinary DATA columns.
### Distance-Axis TAG
```sql
CREATE TAG TABLE ch5_distance_ddl (
name VARCHAR(32) PRIMARY KEY,
distance DOUBLE BASEDISTANCE,
value DOUBLE
);
```
Use `DOUBLE` for fractional distances, or `LONG`/`ULONG` for integer axes according to range.
## Alter a TAG Table
The METADATA ADD/DROP exercise requires Standard Edition. Arbitrary TAG DATA
column changes are restricted. Consider migrating to a new table for schema
expansion. Metadata columns can be added or dropped with supported syntax.
```sql
ALTER TABLE ch5_tag_ddl
METADATA ADD COLUMN (team VARCHAR(32));
ALTER TABLE ch5_tag_ddl METADATA
ADD COLUMN (limits DECIMAL(12,4)[2] DEFAULT [0.0000, NULL]);
ALTER TABLE ch5_tag_ddl
METADATA DROP COLUMN (team);
ALTER TABLE ch5_tag_ddl METADATA
DROP COLUMN (limits);
```
Standard Edition can add fixed-length numeric ARRAY columns to TAG METADATA.
Existing metadata rows receive the specified ARRAY DEFAULT. Rows automatically
registered by TAG DATA input after ALTER do not reapply that DEFAULT; the new
ARRAY column is whole NULL.
Ordinary TAG DATA ARRAY columns can be declared at `CREATE TABLE` but not added
with ALTER. TAG METADATA ARRAY columns have no automatic indexes and do not
support explicit indexes. See [TAG Metadata](../tag-metadata/) and
[Numeric ARRAY Types](/dbms/reference/sql/types/array/) for details.
Before changing a populated production table, check dependent queries, SDK
column order, and reingestion paths. After ALTER, verify with `DESC ch5_tag_ddl;`
and METADATA queries.
## Drop a TAG Table
`DROP TABLE` removes raw data and metadata together. Remove dependent objects
such as ROLLUP in dependency order first. TAG `DROP TABLE ... CASCADE` can also
remove associated ROLLUPs, so do not use it as a routine cleanup default. Check
additional dependency restrictions on custom ROLLUP target tables.
```sql
DROP TABLE ch5_distance_ddl;
DROP TABLE ch5_tag_ddl;
```
See the [DDL Syntax Reference](/dbms/reference/sql/syntax/ddl-syntax/) for exact
properties, supported scope, and DDL.
Read next:
- [TAG Table Structure and Schema](../table-structure-schema/)
- [TAG Data Ingestion and Changes](../data-input-mutation/)
- [TAG Metadata](../tag-metadata/)
---
title: "5.4 Data Ingestion and Changes"
url: https://docs.machbase.com/dbms/tag-table-usage/data-input-mutation/
language: en
kind: page
---
# 5.4 Data Ingestion and Changes
Ingest TAG data with SQL `INSERT`, Append APIs, or file-loading tools. Use SQL examples
to check features and load small amounts; consider Append APIs first for continuous collection.
## SQL INSERT
This example creates time-axis and distance-axis TAG tables, verifies data,
and cleans up. Ensure exercise names do not conflict with existing tables.
Time-axis names identify sensors; distance-axis names identify inspection
subjects or runs.
```sql
CREATE TAG TABLE ch5_input_time (
name VARCHAR(32) PRIMARY KEY,
time DATETIME BASETIME,
value DOUBLE
);
INSERT INTO ch5_input_time
VALUES ('TEMP_001', TO_DATE('2026-01-01 10:00:00', 'YYYY-MM-DD HH24:MI:SS'), 25.5);
INSERT INTO ch5_input_time
VALUES ('TEMP_001', TO_DATE('2026-01-01 10:01:00', 'YYYY-MM-DD HH24:MI:SS'), 25.7);
CREATE TAG TABLE ch5_input_distance (
name VARCHAR(32) PRIMARY KEY,
distance DOUBLE BASEDISTANCE,
value DOUBLE,
quality INTEGER
);
INSERT INTO ch5_input_distance VALUES ('PIPE_A', 0.0, 10.1, 100);
INSERT INTO ch5_input_distance VALUES ('PIPE_A', 500.5, 11.2, 100);
EXEC TABLE_FLUSH(ch5_input_time);
EXEC TABLE_FLUSH(ch5_input_distance);
SELECT name, time, value FROM ch5_input_time ORDER BY time;
SELECT name, distance, value, quality
FROM ch5_input_distance
ORDER BY distance;
SELECT COUNT(*) FROM ch5_input_time;
SELECT COUNT(*) FROM ch5_input_distance;
DROP TABLE ch5_input_distance;
DROP TABLE ch5_input_time;
```
Each table returns two rows. Time-axis values are 25.5 and 25.7; distance-axis
positions are 0.0 and 500.5. Tags were not preregistered, so the first DATA
input registers each name automatically. The application manages actual event
times and retransmission status.
Use `TABLE_FLUSH` in validation or operational procedures that explicitly need
to flush pending storage/input buffers. It is not a transaction commit or a
query-visibility guarantee. Do not execute it per row in ordinary collection
loops. See the [EXEC Procedure Reference](/dbms/reference/sql/syntax/execute-procedure-syntax/#table-flush)
for arguments and errors.
## Ingest with Metadata
A TAG table with user metadata can automatically register new tags through
DATA-only input. If attributes such as location and units must be set first,
register with `INSERT ... METADATA` before DATA input. Supported syntax can
also send DATA and metadata together. Do not assume ordinary DATA input
updates existing attributes every time. Omit system-managed columns from input lists.
See [TAG Metadata](../tag-metadata/) for metadata registration, updates, and deletion.
## Choose an Input Path
| Path | Suitable for | Checks |
| --- | --- | --- |
| SQL `INSERT` | Feature checks, low-frequency input | Per-statement parsing and round-trip cost |
| SDK Append | Continuous high-throughput input | Batch size, flush, error handling |
| `csvimport` / `machloader` | Bulk client-file loads | Column order, date format, bad files |
| `LOAD DATA INFILE` | Server-accessible files | Server paths/permissions, error policy |
See [Development Integration](/dbms/development-tools-integration/) for SDK
connections and Append examples, and
[Data Ingestion and Export](/dbms/development-tools-integration/data-input-load-export/)
for file formats and commands.
## Correct Data
TAG data UPDATE in Standard Edition requires both tag selection and BASETIME
ranges. Tag names, axes, and metadata columns are not ordinary data UPDATE
targets. Existing ROLLUP results do not change automatically after correction;
rebuild the affected range explicitly. See [ROLLUP_REBUILD](../../tag-rollup-usage/rollup-rebuild/).
Check success responses, failure counts, and retry policies in the selected
API. Distinguish SQL NULL, SDK NULL representations, and numeric 0, and define
a duplicate policy for retransmitting observations. For numeric ARRAY and
sparse input, use the
[ARRAY Append Examples](../../development-tools-integration/data-input-load-export/array-append/).
See [TAG Data Correction](../tag-data-update-correction/) for detailed procedures.
---
title: "5.5 Queries and Analysis"
url: https://docs.machbase.com/dbms/tag-table-usage/query-analysis/
language: en
kind: page
---
# 5.5 Queries and Analysis
This page covers common TAG time-series query patterns: time and distance ranges,
multiple tags, and statistics views.
## Query Tag Data
### Sample Schema (Time Axis)
This example registers two tags and inserts 10 rows per tag. `TAG_0001` has data
from January 1–10, 2018; `TAG_0002` has data from February 1–10.
```sql
create tag table TAG (name varchar(20) primary key, time datetime basetime, value double summarized);
insert into tag metadata values ('TAG_0001');
insert into tag metadata values ('TAG_0002');
insert into tag values('TAG_0001', '2018-01-01 01:00:00 000:000:000', 1);
insert into tag values('TAG_0001', '2018-01-02 02:00:00 000:000:000', 2);
insert into tag values('TAG_0001', '2018-01-03 03:00:00 000:000:000', 3);
insert into tag values('TAG_0001', '2018-01-04 04:00:00 000:000:000', 4);
insert into tag values('TAG_0001', '2018-01-05 05:00:00 000:000:000', 5);
insert into tag values('TAG_0001', '2018-01-06 06:00:00 000:000:000', 6);
insert into tag values('TAG_0001', '2018-01-07 07:00:00 000:000:000', 7);
insert into tag values('TAG_0001', '2018-01-08 08:00:00 000:000:000', 8);
insert into tag values('TAG_0001', '2018-01-09 09:00:00 000:000:000', 9);
insert into tag values('TAG_0001', '2018-01-10 10:00:00 000:000:000', 10);
insert into tag values('TAG_0002', '2018-02-01 01:00:00 000:000:000', 11);
insert into tag values('TAG_0002', '2018-02-02 02:00:00 000:000:000', 12);
insert into tag values('TAG_0002', '2018-02-03 03:00:00 000:000:000', 13);
insert into tag values('TAG_0002', '2018-02-04 04:00:00 000:000:000', 14);
insert into tag values('TAG_0002', '2018-02-05 05:00:00 000:000:000', 15);
insert into tag values('TAG_0002', '2018-02-06 06:00:00 000:000:000', 16);
insert into tag values('TAG_0002', '2018-02-07 07:00:00 000:000:000', 17);
insert into tag values('TAG_0002', '2018-02-08 08:00:00 000:000:000', 18);
insert into tag values('TAG_0002', '2018-02-09 09:00:00 000:000:000', 19);
insert into tag values('TAG_0002', '2018-02-10 10:00:00 000:000:000', 20);
exec table_flush(tag);
```
The final `TABLE_FLUSH` explicitly processes storage buffers. Do not interpret it
as a transaction commit or a guarantee of query visibility. See
[TABLE_FLUSH](/dbms/reference/sql/syntax/execute-procedure-syntax/#table-flush) for details.
### Retrieve All TAG Data
```sql
select * from tag ORDER BY name, time;
```
```text
NAME TIME VALUE
--------------------------------------------------------------------------------------
TAG_0001 2018-01-01 01:00:00 000:000:000 1
TAG_0001 2018-01-02 02:00:00 000:000:000 2
TAG_0001 2018-01-03 03:00:00 000:000:000 3
TAG_0001 2018-01-04 04:00:00 000:000:000 4
TAG_0001 2018-01-05 05:00:00 000:000:000 5
TAG_0001 2018-01-06 06:00:00 000:000:000 6
TAG_0001 2018-01-07 07:00:00 000:000:000 7
TAG_0001 2018-01-08 08:00:00 000:000:000 8
TAG_0001 2018-01-09 09:00:00 000:000:000 9
TAG_0001 2018-01-10 10:00:00 000:000:000 10
TAG_0002 2018-02-01 01:00:00 000:000:000 11
TAG_0002 2018-02-02 02:00:00 000:000:000 12
TAG_0002 2018-02-03 03:00:00 000:000:000 13
TAG_0002 2018-02-04 04:00:00 000:000:000 14
TAG_0002 2018-02-05 05:00:00 000:000:000 15
TAG_0002 2018-02-06 06:00:00 000:000:000 16
TAG_0002 2018-02-07 07:00:00 000:000:000 17
TAG_0002 2018-02-08 08:00:00 000:000:000 18
TAG_0002 2018-02-09 09:00:00 000:000:000 19
TAG_0002 2018-02-10 10:00:00 000:000:000 20
[20] row(s) selected.
```
The output is one execution result. Specify `ORDER BY name, time` when ordering
must be guaranteed. Unfiltered output order may depend on the execution plan and
scan direction.
### Retrieve Data by Tag Name
This example retrieves data for TAG_0002.
```sql
select * from tag where name='TAG_0002' ORDER BY name, time;
```
```text
NAME TIME VALUE
--------------------------------------------------------------------------------------
TAG_0002 2018-02-01 01:00:00 000:000:000 11
TAG_0002 2018-02-02 02:00:00 000:000:000 12
TAG_0002 2018-02-03 03:00:00 000:000:000 13
TAG_0002 2018-02-04 04:00:00 000:000:000 14
TAG_0002 2018-02-05 05:00:00 000:000:000 15
TAG_0002 2018-02-06 06:00:00 000:000:000 16
TAG_0002 2018-02-07 07:00:00 000:000:000 17
TAG_0002 2018-02-08 08:00:00 000:000:000 18
TAG_0002 2018-02-09 09:00:00 000:000:000 19
TAG_0002 2018-02-10 10:00:00 000:000:000 20
[10] row(s) selected.
```
### Query a Time Range
This example applies a time range to TAG_0002.
> `BETWEEN` includes both boundaries, equivalent to `>=` together with `<=`.
> The examples return the same results with `>` and `<` because there are no rows
> at the boundary timestamps. To avoid reading boundary rows twice in adjacent
> intervals, use `time >= start AND time < end`.
```sql
select * from tag where name = 'TAG_0002' and time between to_date('2018-02-01') and to_date('2018-02-05') ORDER BY name, time;
```
```text
NAME TIME VALUE
--------------------------------------------------------------------------------------
TAG_0002 2018-02-01 01:00:00 000:000:000 11
TAG_0002 2018-02-02 02:00:00 000:000:000 12
TAG_0002 2018-02-03 03:00:00 000:000:000 13
TAG_0002 2018-02-04 04:00:00 000:000:000 14
[4] row(s) selected.
```
```sql
select * from tag where name = 'TAG_0002' and time > to_date('2018-02-01') and time < to_date('2018-02-05') ORDER BY name, time;
```
```text
NAME TIME VALUE
--------------------------------------------------------------------------------------
TAG_0002 2018-02-01 01:00:00 000:000:000 11
TAG_0002 2018-02-02 02:00:00 000:000:000 12
TAG_0002 2018-02-03 03:00:00 000:000:000 13
TAG_0002 2018-02-04 04:00:00 000:000:000 14
[4] row(s) selected.
```
### Distance-Axis Sample Schema
This example uses a distance-axis (`BASE DISTANCE`) TAG table.
```sql
CREATE TAG TABLE trip_tag (
name VARCHAR(20) PRIMARY KEY,
distance_m DOUBLE BASE DISTANCE,
value DOUBLE,
quality INTEGER
);
INSERT INTO trip_tag VALUES('ODO_A', 0, 10.1, 100);
INSERT INTO trip_tag VALUES('ODO_A', 500, 11.2, 101);
INSERT INTO trip_tag VALUES('ODO_A', 1000, 12.3, 102);
INSERT INTO trip_tag VALUES('ODO_B', 1000.1, 21.5, 100);
INSERT INTO trip_tag VALUES('ODO_B', 1500, 22.1, 101);
INSERT INTO trip_tag VALUES('ODO_B', 2000, 22.9, 102);
EXEC TABLE_FLUSH(trip_tag);
```
### Query a Distance Range
```sql
SELECT name, distance_m, value, quality
FROM trip_tag
WHERE name = 'ODO_A'
AND distance_m BETWEEN 0 AND 1000
ORDER BY distance_m;
```
As with a time axis, narrowing the axis range is the basic query pattern.
### Fractional Boundaries on a DOUBLE Distance Axis
```sql
SELECT name, distance_m, value, quality
FROM trip_tag
WHERE name = 'ODO_B'
AND distance_m BETWEEN 1000.1 AND 2000
ORDER BY distance_m;
```
Numeric comparison excludes `1000` and includes `1500` and `2000`.
### Check Distance-Axis Execution Plans
For large distance-range queries, use `EXPLAIN` to check whether distance predicates
become key ranges.
```sql
EXPLAIN
SELECT name, distance_m, value
FROM trip_tag
WHERE name = 'ODO_B'
AND distance_m BETWEEN 1000.1 AND 2000
ORDER BY distance_m;
```
Check for:
- `KEYVALUE INDEX SCAN` or a similar index scan
- A `distance_m between ...` condition under `KEY RANGE`
### Aggregate into Distance Buckets
This example groups nonnegative distances into intervals of 500. TRUNC truncates
toward zero. For negative coordinates, choose the desired boundary rule separately,
such as FLOOR.
```sql
SELECT TRUNC(distance_m / 500, 0) * 500 AS dist_bucket,
COUNT(*) AS sample_count,
MIN(value) AS min_v,
MAX(value) AS max_v,
AVG(value) AS avg_v
FROM trip_tag
WHERE name = 'ODO_B'
GROUP BY TRUNC(distance_m / 500, 0) * 500
ORDER BY dist_bucket;
```
For example, `1750` belongs to bucket `1500`.
### Query a Time Range Across Multiple Tags
Apply the same time range to two or more tags. Use `IN` when the target name list
is known. For many tags, measure performance with both list size and time-range width.
```sql
select * from tag where name in ('TAG_0002', 'TAG_0001') and time between to_date('2018-01-05') and to_date('2018-02-05') ORDER BY name, time;
```
```text
NAME TIME VALUE
--------------------------------------------------------------------------------------
TAG_0001 2018-01-05 05:00:00 000:000:000 5
TAG_0001 2018-01-06 06:00:00 000:000:000 6
TAG_0001 2018-01-07 07:00:00 000:000:000 7
TAG_0001 2018-01-08 08:00:00 000:000:000 8
TAG_0001 2018-01-09 09:00:00 000:000:000 9
TAG_0001 2018-01-10 10:00:00 000:000:000 10
TAG_0002 2018-02-01 01:00:00 000:000:000 11
TAG_0002 2018-02-02 02:00:00 000:000:000 12
TAG_0002 2018-02-03 03:00:00 000:000:000 13
TAG_0002 2018-02-04 04:00:00 000:000:000 14
[10] row(s) selected.
```
### Filter by Values
You can also filter tag values. This example selects TAG_0002 values greater than
12 and less than 15.
```sql
select * from tag where name = 'TAG_0002' and value > 12 and value < 15 and time between to_date('2018-02-01') and to_date('2018-02-05') ORDER BY name, time;
```
```text
NAME TIME VALUE
--------------------------------------------------------------------------------------
TAG_0002 2018-02-03 03:00:00 000:000:000 13
TAG_0002 2018-02-04 04:00:00 000:000:000 14
[2] row(s) selected.
```
### Per-Tag Statistics View `V$
_STAT`
Creating a TAG table automatically creates a virtual table aggregating statistics
per tag ID, named v${tag table name}_stat.
Distinguish tag-name/axis statistics from value statistics for the third SUMMARIZED
column. STAT reflects background index/statistics processing, so verify recent
input with source SELECT queries as well. Exercises needing immediate statistics
use TABLE_FLUSH followed by INDEX_FLUSH to wait for processing.
Axis-specific BASE DISTANCE STAT schemas are supported since Machbase 8.7.0
Axis-related column names and types depend on the TAG axis.
| TAG axis | Minimum/maximum axis | Axis of minimum/maximum value | Latest inserted row axis | Axis statistic type |
|--------|--------------|----------------------|------------------|--------------|
| `DATETIME BASE TIME` | `MIN_TIME`, `MAX_TIME` | `MIN_VALUE_TIME`, `MAX_VALUE_TIME` | `RECENT_ROW_TIME` | `DATETIME` |
| `DOUBLE/LONG/ULONG BASE DISTANCE` | `MIN_DISTANCE`, `MAX_DISTANCE` | `MIN_VALUE_DISTANCE`, `MAX_VALUE_DISTANCE` | `RECENT_ROW_DISTANCE` | Original BASE DISTANCE type |
Both Editions share `NAME`, `ROW_COUNT`, `MIN_VALUE`, and `MAX_VALUE`. Cluster
Edition adds `HOSTNAME VARCHAR(64)` at the beginning of the schema.
#### BASE TIME STAT Schema
```sql
DESC v$tag_stat;
```
```text
[ COLUMN ]
----------------------------------------------------------------------------------------------------
NAME NULL? TYPE LENGTH
----------------------------------------------------------------------------------------------------
NAME varchar 100
ROW_COUNT ulong 20
MIN_TIME datetime 31
MAX_TIME datetime 31
MIN_VALUE double 17
MIN_VALUE_TIME datetime 31
MAX_VALUE double 17
MAX_VALUE_TIME datetime 31
RECENT_ROW_TIME datetime 31
```
Without SUMMARIZED on the third column, VALUE statistics (MIN_VALUE, MAX_VALUE,
MIN_VALUE_TIME, MAX_VALUE_TIME) are not stored.
Collected statistics:
| Column | Information |
|--|--|
| NAME | Tag ID name |
| ROW_COUNT | Row count |
| MIN_TIME | Minimum basetime for this tag ID |
| MAX_TIME | Maximum basetime for this tag ID |
| MIN_VALUE | Minimum summarized value for this tag ID |
| MIN_VALUE_TIME | Basetime inserted with MIN_VALUE |
| MAX_VALUE | Maximum summarized value for this tag ID |
| MAX_VALUE_TIME | Basetime inserted with MAX_VALUE |
| RECENT_ROW_TIME | Basetime of the most recently inserted row |
The statistics exercise uses a separate table from the preceding 20-row query
example. Only its two tags appear; TAG_0001 and TAG_0002 do not affect expected statistics.
1. With a SUMMARIZED column
```sql
CREATE TAG TABLE ch5_stat_time (name VARCHAR(20) PRIMARY KEY, time DATETIME BASETIME, value DOUBLE SUMMARIZED);
```
```sql
INSERT INTO ch5_stat_time VALUES('tag-0', TO_DATE('2021-08-12'), 10);
INSERT INTO ch5_stat_time VALUES('tag-0', TO_DATE('2021-08-13'), 10);
INSERT INTO ch5_stat_time VALUES('tag-0', TO_DATE('2021-08-14'), 20);
INSERT INTO ch5_stat_time VALUES('tag-0', TO_DATE('2021-08-11'), 5);
INSERT INTO ch5_stat_time VALUES('tag-1', TO_DATE('2022-08-12'), 100);
INSERT INTO ch5_stat_time VALUES('tag-1', TO_DATE('2022-08-11'), 200);
INSERT INTO ch5_stat_time VALUES('tag-1', TO_DATE('2022-08-10'), 50);
```
```sql
EXEC TABLE_FLUSH(ch5_stat_time);
EXEC INDEX_FLUSH(ch5_stat_time);
SELECT * FROM v$ch5_stat_time_stat ORDER BY name;
```
```text
NAME ROW_COUNT MIN_TIME MAX_TIME MIN_VALUE
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
MIN_VALUE_TIME MAX_VALUE MAX_VALUE_TIME RECENT_ROW_TIME
---------------------------------------------------------------------------------------------------------------------------------
tag-0 4 2021-08-11 00:00:00 000:000:000 2021-08-14 00:00:00 000:000:000 5
2021-08-11 00:00:00 000:000:000 20 2021-08-14 00:00:00 000:000:000 2021-08-11 00:00:00 000:000:000
tag-1 3 2022-08-10 00:00:00 000:000:000 2022-08-12 00:00:00 000:000:000 50
2022-08-10 00:00:00 000:000:000 200 2022-08-11 00:00:00 000:000:000 2022-08-10 00:00:00 000:000:000
[2] row(s) selected.
```
2. Without a SUMMARIZED column
```sql
CREATE TAG TABLE other_tag (name VARCHAR(20) PRIMARY KEY, time DATETIME BASETIME, value DOUBLE);
```
```text
Executed successfully.
```
```sql
INSERT INTO other_tag VALUES('tag-0', TO_DATE('2021-08-12'), 10);
INSERT INTO other_tag VALUES('tag-0', TO_DATE('2021-08-13'), 10);
INSERT INTO other_tag VALUES('tag-0', TO_DATE('2021-08-14'), 20);
INSERT INTO other_tag VALUES('tag-0', TO_DATE('2021-08-11'), 5);
INSERT INTO other_tag VALUES('tag-1', TO_DATE('2022-08-12'), 100);
INSERT INTO other_tag VALUES('tag-1', TO_DATE('2022-08-11'), 200);
INSERT INTO other_tag VALUES('tag-1', TO_DATE('2022-08-10'), 50);
```
```sql
EXEC TABLE_FLUSH(other_tag);
EXEC INDEX_FLUSH(other_tag);
SELECT * FROM v$other_tag_stat ORDER BY name;
```
```text
NAME ROW_COUNT MIN_TIME MAX_TIME MIN_VALUE
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
MIN_VALUE_TIME MAX_VALUE MAX_VALUE_TIME RECENT_ROW_TIME
---------------------------------------------------------------------------------------------------------------------------------
tag-0 4 2021-08-11 00:00:00 000:000:000 2021-08-14 00:00:00 000:000:000 NULL
NULL NULL NULL 2021-08-11 00:00:00 000:000:000
tag-1 3 2022-08-10 00:00:00 000:000:000 2022-08-12 00:00:00 000:000:000 NULL
NULL NULL NULL 2022-08-10 00:00:00 000:000:000
[2] row(s) selected.
```
#### BASE DISTANCE STAT Schema
Distance-axis TAG statistics views expose distance values as numeric types.
```sql
CREATE TAG TABLE distance_sensor (
name VARCHAR(32) PRIMARY KEY,
odometer_m DOUBLE BASE DISTANCE,
value DOUBLE SUMMARIZED
);
INSERT INTO distance_sensor VALUES('sensor', 20.5, 8);
INSERT INTO distance_sensor VALUES('sensor', 10.25, 3);
INSERT INTO distance_sensor VALUES('sensor', 30.75, 5);
EXEC TABLE_FLUSH(distance_sensor);
EXEC INDEX_FLUSH(distance_sensor);
```
In Standard Edition, a `DOUBLE BASE DISTANCE` table has this schema:
```sql
DESC V$DISTANCE_SENSOR_STAT;
```
```text
[ COLUMN ]
----------------------------------------------------------------------------------------------------
NAME NULL? TYPE LENGTH
----------------------------------------------------------------------------------------------------
NAME varchar 100
ROW_COUNT ulong 20
MIN_DISTANCE double 17
MAX_DISTANCE double 17
MIN_VALUE double 17
MIN_VALUE_DISTANCE double 17
MAX_VALUE double 17
MAX_VALUE_DISTANCE double 17
RECENT_ROW_DISTANCE double 17
```
The five distance statistic columns use the original BASE DISTANCE type.
| BASE DISTANCE type | STAT column type | `DESC` length |
|--------------------|----------------|-------------|
| `DOUBLE` | `double` | 17 |
| `LONG` | `long` | 20 |
| `ULONG` | `ulong` | 20 |
```sql
SELECT name,
row_count,
min_distance,
max_distance,
min_value,
min_value_distance,
max_value,
max_value_distance,
recent_row_distance
FROM V$DISTANCE_SENSOR_STAT
WHERE name = 'sensor';
```
- `MIN_DISTANCE` and `MAX_DISTANCE` are the minimum and maximum distances for the statistics row.
- `MIN_VALUE_DISTANCE` and `MAX_VALUE_DISTANCE` are the distances where the minimum
and maximum summarized values occurred.
- `RECENT_ROW_DISTANCE` is the most recently inserted row's distance, not the largest distance.
- Without `SUMMARIZED`, `MIN_VALUE_DISTANCE` and `MAX_VALUE_DISTANCE` are `NULL`.
##### Query in Cluster Edition
Cluster statistics views add `HOSTNAME` and may return rows per Warehouse. Inspect
Warehouse-specific values first.
```sql
SELECT hostname, name, row_count,
min_distance, max_distance,
min_value, min_value_distance,
max_value, max_value_distance,
recent_row_distance
FROM V$DISTANCE_SENSOR_STAT
ORDER BY hostname, name;
```
Row counts and distance boundaries can be safely aggregated by tag name.
```sql
SELECT name,
SUM(row_count) AS row_count,
MIN(min_distance) AS min_distance,
MAX(max_distance) AS max_distance
FROM V$DISTANCE_SENSOR_STAT
GROUP BY name;
```
{{< callout type="warning" >}}
Keep `MIN_VALUE` paired with `MIN_VALUE_DISTANCE` and `MAX_VALUE` with
`MAX_VALUE_DISTANCE` from the same Warehouse row. Independent `MIN` or `MAX`
aggregations can combine values from different Warehouses. `RECENT_ROW_DISTANCE`
is also per-Warehouse; do not interpret `MAX(RECENT_ROW_DISTANCE)` as the latest
inserted row across the cluster.
{{< /callout >}}
##### 8.7.0 Compatibility
Old BASE DISTANCE statistics column names are not provided as aliases. Existing
tables also use the new schema after the 8.7.0 server restarts.
| Before 8.7.0 | 8.7.0 |
|-----------------|------------|
| `MIN_TIME` | `MIN_DISTANCE` |
| `MAX_TIME` | `MAX_DISTANCE` |
| `MIN_VALUE_TIME` | `MIN_VALUE_DISTANCE` |
| `MAX_VALUE_TIME` | `MAX_VALUE_DISTANCE` |
| `RECENT_ROW_TIME` | `RECENT_ROW_DISTANCE` |
BASE TIME TAG tables retain the `*_TIME DATETIME` schema.
### Scan Direction Hints
Distinguish axis traversal from result sorting. The latest value in a reverse-axis
scan is the largest axis value; it may differ from STAT RECENT_ROW, which describes
the last inserted row. Ordering multiple rows with the same axis value requires
an additional application-defined criterion.
```sql
SELECT *
FROM tag
WHERE name = 'TAG_0001'
ORDER BY time
LIMIT 10;
SELECT /*+ SCAN_FORWARD(tag) */ name, time, value
FROM tag
WHERE name = 'TAG_0001'
LIMIT 10;
SELECT /*+ SCAN_BACKWARD(tag) */ name, time, value
FROM tag
WHERE name = 'TAG_0001'
LIMIT 10;
```
See [TABLE_SCAN_DIRECTION](/dbms/reference/configuration/configuration/) for the
default direction without a hint.
## Cleanup
```sql
DROP TABLE ch5_stat_time;
DROP TABLE distance_sensor;
DROP TABLE trip_tag;
DROP TABLE other_tag;
DROP TABLE tag;
```
---
title: "5.6 Indexes and Performance"
url: https://docs.machbase.com/dbms/tag-table-usage/index-performance/
language: en
kind: page
---
# 5.6 Indexes and Performance
Start TAG queries by limiting tag names and axis ranges. Choose additional indexes only
after measuring actual predicates and execution plans.
## Basic Query Paths
TAG tables automatically manage structures for queries by `PRIMARY KEY` tag
name and `BASETIME` or `BASEDISTANCE`. Applications should not depend on
generated system-object names or storage stages.
| Query condition | Tuning direction |
| --- | --- |
| Axis range for one tag | Specify both tag name and axis range |
| Same time range for multiple tags | Limit time first and manage target tag count |
| Metadata attributes | Define TAG `METADATA` columns |
| Repeated time aggregation | Consider ROLLUP |
| Value-driven queries | Validate secondary value indexes with execution plans |
Broad queries without tag or axis limits read more data. Check `EXPLAIN` and
execution time with actual volumes instead of assuming constant speed.
## METADATA Columns
Use `METADATA` for attributes defined once per tag, such as location or device
type. Ordinary scalar METADATA columns have automatic search indexes. Raw
JSON columns do not; index required JSON paths explicitly. Numeric ARRAY
metadata supports neither automatic nor explicit indexes. See [TAG Metadata](../tag-metadata/).
Putting time-series values or frequently changing status in METADATA obscures
update semantics. Depending on the value, consider TAG DATA columns, LOOKUP,
VOLATILE, or TRANSACTION tables (Standard Edition only).
## Secondary Indexes on Value Columns
For frequent value predicates, consider `INDEX_TYPE TAG` secondary indexes.
They increase ingestion and storage costs; compare representative queries
before and after creation.
This example uses isolated names to create value and JSON path indexes,
checks execution plans, and removes all objects. The small sample validates
syntax and results, not performance benefits. Run in a separate environment
without name conflicts.
```sql
CREATE TAG TABLE ch5_index_tag (
name VARCHAR(32) PRIMARY KEY,
time DATETIME BASETIME,
value DOUBLE,
payload JSON
) METADATA (
location VARCHAR(64)
);
INSERT INTO ch5_index_tag METADATA VALUES ('TEMP-01', 'LINE-A');
INSERT INTO ch5_index_tag METADATA VALUES ('TEMP-02', 'LINE-B');
INSERT INTO ch5_index_tag VALUES
('TEMP-01', TO_DATE('2026-01-01 00:00:00', 'YYYY-MM-DD HH24:MI:SS'),
10.0, '{"state":"normal"}');
INSERT INTO ch5_index_tag VALUES
('TEMP-01', TO_DATE('2026-01-01 00:01:00', 'YYYY-MM-DD HH24:MI:SS'),
90.0, '{"state":"alarm"}');
INSERT INTO ch5_index_tag VALUES
('TEMP-02', TO_DATE('2026-01-01 00:02:00', 'YYYY-MM-DD HH24:MI:SS'),
95.0, '{"state":"alarm"}');
CREATE INDEX idx_ch5_index_tag_value
ON ch5_index_tag (value) INDEX_TYPE TAG;
CREATE INDEX idx_ch5_index_tag_json
ON ch5_index_tag (payload->'$.state');
EXPLAIN SELECT name, time, value
FROM ch5_index_tag
WHERE name = 'TEMP-01'
AND time BETWEEN TO_DATE('2026-01-01', 'YYYY-MM-DD')
AND TO_DATE('2026-01-02', 'YYYY-MM-DD')
AND value > 80.0;
SELECT name, value FROM ch5_index_tag
WHERE location = 'LINE-A' AND value > 80.0
ORDER BY name, time;
SELECT name, value FROM ch5_index_tag
WHERE payload->'$.state' = 'alarm'
ORDER BY name, time;
DROP INDEX idx_ch5_index_tag_json;
DROP INDEX idx_ch5_index_tag_value;
DROP TABLE ch5_index_tag;
```
The first SELECT returns TEMP-01 with 90.0; the second returns TEMP-01 with
90.0 and TEMP-02 with 95.0. Verify identical results before and after indexing.
With production-scale data, compare ingestion cost and index size as well as query time.
Match JSON path result types to comparison value types, and use `EXPLAIN`
to check whether the intended path uses an index.
## Avoid Inapplicable Tuning
- Do not create separate indexes on TAG axis columns.
- Do not apply the LOG `MINMAX_CACHE_SIZE` setting to TAG value columns.
- Consider ROLLUP for repeated broad-period aggregation instead of relying
only on secondary indexes.
See the [SQL Syntax Reference](/dbms/reference/sql/syntax/) for index syntax
and [Query Tuning](/dbms/performance-tuning/performance-query-tuning/) for
measurement and tuning procedures.
---
title: "5.7 Operations and Data Lifecycle"
url: https://docs.machbase.com/dbms/tag-table-usage/operations-lifecycle/
language: en
kind: page
---
# 5.7 Operations and Data Lifecycle
Manage TAG data through manual deletion, retention policies, and duplicate prevention.
This page explains operational choices and links to complete SQL references.
## Delete TAG Data
Limit deletion with tag identifiers and axis predicates. Common time-axis
TAG choices are:
| Purpose | Condition |
| --- | --- |
| Delete all data for one tag | Match tag `PRIMARY KEY` |
| Delete one tag's time range | Tag match + BASETIME range |
| Delete old data for all tags | BASETIME predicate or `BEFORE` |
| Delete all table data | TAG `DELETE` without predicates |
This example creates a separate test table, checks deletion scope, and cleans up.
```sql
CREATE TAG TABLE ch5_lifecycle (
name VARCHAR(32) PRIMARY KEY,
time DATETIME BASETIME,
value DOUBLE
);
INSERT INTO ch5_lifecycle VALUES
('TAG_0001', TO_DATE('2026-01-01 00:00:00', 'YYYY-MM-DD HH24:MI:SS'), 1.0);
INSERT INTO ch5_lifecycle VALUES
('TAG_0001', TO_DATE('2026-01-02 00:00:00', 'YYYY-MM-DD HH24:MI:SS'), 2.0);
INSERT INTO ch5_lifecycle VALUES
('TAG_0002', TO_DATE('2026-01-01 00:00:00', 'YYYY-MM-DD HH24:MI:SS'), 3.0);
DELETE FROM ch5_lifecycle
WHERE name = 'TAG_0001'
AND time < TO_DATE('2026-01-02 00:00:00', 'YYYY-MM-DD HH24:MI:SS');
SELECT name, time, value
FROM ch5_lifecycle
ORDER BY name, time;
DELETE FROM ch5_lifecycle;
SELECT COUNT(*) FROM ch5_lifecycle;
DROP TABLE ch5_lifecycle;
```
After the first deletion, TAG_0001 retains value 2.0 at 2026-01-02, and
TAG_0002 retains 3.0. After full deletion, COUNT is 0. DATA and METADATA
deletion are separate. To remove tag registrations, also check
[METADATA Deletion](../tag-metadata/) requirements.
See [TAG DELETE Syntax](/dbms/reference/sql/syntax/dml-syntax/) for operators
and Edition support. Use [Retention Policies](/dbms/operations-configuration-recovery/policy-data-retention/)
when deletion must run periodically.
### Handle ROLLUP Data
Deleting raw TAG data and handling existing ROLLUP are separate operations.
If aggregates must reflect corrected or deleted source data, rebuild the
affected ROLLUP range. Do not repeatedly execute ROLLUP deletion as a
retention policy.
- [Partial ROLLUP Deletion and Rebuild](/dbms/tag-rollup-usage/rollup-rebuild/)
- [Rebuild ROLLUP After TAG Correction](../tag-data-update-correction/)
## Automatic Deduplication
`TAG_DUPLICATE_CHECK_DURATION` sets the duplicate-check interval in minutes.
Standard Edition accepts 0–43200; 0 disables it. Cluster accepts only 0,
so the following enablement exercise is Standard Edition only.
Within the interval measured against server time, rows with identical tags,
axis values, and DATA values are duplicates. Detection occurs during index
processing and duplicate cleanup, so an Append success count is not an
already-deduplicated count. Late data outside the interval may not be
deduplicated as expected. This does not replace business-key uniqueness constraints.
```sql
CREATE TAG TABLE ch5_dedup (
name VARCHAR(32) PRIMARY KEY,
time DATETIME BASETIME,
value DOUBLE
) TAG_DUPLICATE_CHECK_DURATION = 1440;
INSERT INTO ch5_dedup VALUES ('TAG_0001', NOW, 1.0);
INSERT INTO ch5_dedup SELECT name, time, value FROM ch5_dedup;
EXEC TABLE_FLUSH(ch5_dedup);
EXEC INDEX_FLUSH(ch5_dedup);
SELECT name, time, value
FROM ch5_dedup
WHERE name = 'TAG_0001';
ALTER TABLE ch5_dedup SET TAG_DUPLICATE_CHECK_DURATION = 60;
DROP TABLE ch5_dedup;
```
The second input copies the first row, including its exact axis timestamp.
This exercise assumes no concurrent production ingestion. Wait for storage
buffers and index processing, then verify that one row remains. Do not call
both flush procedures per row in a normal collection loop.
Check full properties and restrictions against the current
[CREATE TAG TABLE Syntax](/dbms/reference/sql/syntax/ddl-syntax/). Data already
deleted by retention is no longer available for duplicate comparison.
## Operational Checklist
1. Define raw and ROLLUP retention periods separately.
2. Measure maximum arrival delay to choose a duplicate-check interval.
3. Verify target tags and time ranges with SELECT before deleting or correcting.
4. Validate ROLLUP and representative queries after bulk changes.
5. Monitor input volume, disk use, and retention execution together.
---
title: "5.8 Constraints, Errors, and Troubleshooting"
url: https://docs.machbase.com/dbms/tag-table-usage/constraints-errors-troubleshooting/
language: en
kind: page
---
# 5.8 Constraints, Errors, and Troubleshooting
The UPDATE exercises require Standard Edition. Create these tables first and run
valid SQL separately from intentional failures. Do not include failure examples
in a normal script. Do not delete existing objects with conflicting names.
```sql
CREATE TAG TABLE ch5_error_time (
name VARCHAR(64) PRIMARY KEY,
time DATETIME BASETIME,
value DOUBLE,
status INTEGER
) METADATA (location VARCHAR(64));
INSERT INTO ch5_error_time METADATA VALUES ('sensor-01', 'zone-1');
INSERT INTO ch5_error_time METADATA VALUES ('sensor-02', 'zone-2');
INSERT INTO ch5_error_time VALUES
('sensor-01', TO_DATE('2026-07-01 12:00:00', 'YYYY-MM-DD HH24:MI:SS'), 10.0, 0);
INSERT INTO ch5_error_time VALUES
('sensor-02', TO_DATE('2026-07-01 12:00:00', 'YYYY-MM-DD HH24:MI:SS'), 20.0, 0);
CREATE TAG TABLE ch5_error_distance (
name VARCHAR(64) PRIMARY KEY,
distance DOUBLE BASEDISTANCE,
value DOUBLE SUMMARIZED
);
```
## TAG Data UPDATE WHERE Errors
WHERE must include both tag selection and BASETIME predicates. Ambiguous or
unsupported conditions cause UPDATE to fail.
{{< callout type="warning" >}}
**Required Predicates**
Specify both a tag predicate such as `WHERE name ...` and a BASETIME predicate
such as `time ...`. Full UPDATE without conditions, tag-only UPDATE, and
time-only UPDATE are not allowed.
{{< /callout >}}
### Symptom
The following UPDATE statements are rejected:
```sql
-- No time predicate
UPDATE ch5_error_time SET value = 99.9
WHERE name = 'sensor-01';
-- No tag predicate
UPDATE ch5_error_time SET value = 99.9
WHERE time >= TO_DATE('2026-07-01', 'YYYY-MM-DD');
-- Uses OR
UPDATE ch5_error_time SET value = 99.9
WHERE name = 'sensor-01'
OR name = 'sensor-02';
```
### Cause
Conditions that cannot clearly restrict target tags and the time range are rejected.
| Predicate | Supported |
|-----------|:--------:|
| `name = 'sensor-01' AND time >= ...` | O |
| `name IN ('sensor-01', 'sensor-02') AND time BETWEEN ...` | O |
| `name LIKE 'sensor-%' AND time < ...` | O |
| Only `value > 10` | X |
| Only `name = 'sensor-01'` | X |
| Only `time >= ...` | X |
| `OR`, subqueries, aggregate predicates | X |
### Resolution
Specify both tag and time predicates.
```sql
UPDATE ch5_error_time
SET value = 99.9
WHERE name = 'sensor-01'
AND time = TO_DATE('2026-07-01 12:00:00', 'YYYY-MM-DD HH24:MI:SS');
UPDATE ch5_error_time
SET status = 1
WHERE name IN ('sensor-01', 'sensor-02')
AND time >= TO_DATE('2026-07-01 00:00:00', 'YYYY-MM-DD HH24:MI:SS')
AND time < TO_DATE('2026-07-02 00:00:00', 'YYYY-MM-DD HH24:MI:SS');
```
### NAME or TIME Binding Rejected with `ERR-02190`
Since Machbase 8.7.0, Standard Edition supports bind parameters for NAME and
BASETIME predicate values as follows:
```sql
UPDATE ch5_error_time
SET value = ?
WHERE name = ?
AND time = ?;
```
If both required predicates are present but the statement fails with
`ERR-02190: Invalid UPDATE/DELETE condition. Specify it as (primary key column) = (value)`,
check the server version. Older servers do not support NAME/TIME binding for TAG
data UPDATE. Upgrade to 8.7.0 or later. For named markers, also use an 8.7.0 SDK
that supports the named API.
The `?` example is SQL for SDK prepare/bind, not a statement to run unchanged
without values in machsql. When reusing a prepared statement, rebind all SET,
NAME, and TIME values. See
[TAG Data UPDATE Binding](/dbms/reference/sql/syntax/dml-syntax/tag-data-update-syntax/#tag-data-update-predicate-bind)
for marker rules.
Before a large UPDATE, run `SELECT COUNT(*)` with the same WHERE condition to
check the target row count.
## TAG Data UPDATE SET Target Errors
SET can target only actual DATA columns. PRIMARY KEY, BASETIME, and metadata
columns raise errors as SET targets.
{{< callout type="warning" >}}
**SET Targets**
`value` and user DATA columns can be updated. `name`, `time`, and columns in the
METADATA block cannot be SET targets in TAG data UPDATE.
{{< /callout >}}
### Symptom
```sql
-- Error: attempts to update PRIMARY KEY column (name)
UPDATE ch5_error_time
SET name = 'new-sensor'
WHERE name = 'old-sensor'
AND time >= TO_DATE('2026-07-01', 'YYYY-MM-DD');
-- Error: attempts to update BASETIME column (time)
UPDATE ch5_error_time
SET time = NOW
WHERE name = 'sensor-01'
AND time >= TO_DATE('2026-07-01', 'YYYY-MM-DD');
-- Error: changes metadata through data UPDATE
UPDATE ch5_error_time
SET location = 'zone-2'
WHERE name = 'sensor-01'
AND time >= TO_DATE('2026-07-01', 'YYYY-MM-DD');
```
### UPDATE Support by Column Type
| Column type | Description | Data UPDATE |
|-----------|------|:-----------:|
| PRIMARY KEY (`name`) | Unique tag identifier | X |
| BASETIME (`time`) | Time-series timestamp | X |
| DATA (`value`, auxiliary columns) | Actual row values | O |
| `SUMMARIZED` DATA column | Column with statistics | O |
| Metadata column | Tag attribute in METADATA | X |
### Resolution
Use ordinary UPDATE for DATA values.
```sql
UPDATE ch5_error_time
SET value = 99.9,
status = 1
WHERE name = 'sensor-01'
AND time = TO_DATE('2026-07-01 12:00:00', 'YYYY-MM-DD HH24:MI:SS');
```
Use separate syntax for metadata.
```sql
UPDATE ch5_error_time METADATA
SET location = 'zone-2'
WHERE name = 'sensor-01';
```
To change a tag name or time axis, insert data with the new `name`/`time`, then
delete the original data according to operational policy.
## BASE DISTANCE TAG STAT Column Errors
Machbase 8.7.0 exposes distance-named, numeric axis columns in BASE DISTANCE TAG
`V$
_STAT` views. Pre-upgrade SQL referencing `MIN_TIME`, `MAX_TIME`, or
`RECENT_ROW_TIME` may fail with column-not-found errors.
### Symptoms
- Existing `*_TIME` columns cannot be queried in BASE DISTANCE statistics after upgrade.
- Older servers may interpret distances as `DATETIME`, displaying meaningless dates.
- Using Standard ordinal result mappings in Cluster may shift columns because
`HOSTNAME` is added first.
### Diagnosis
Check both the table axis and actual statistics-view schema.
```sql
DESC ch5_error_distance;
DESC V$CH5_ERROR_DISTANCE_STAT;
```
For BASE DISTANCE, `MIN_DISTANCE`, `MAX_DISTANCE`, `MIN_VALUE_DISTANCE`,
`MAX_VALUE_DISTANCE`, and `RECENT_ROW_DISTANCE` should use the original distance
type. Cluster adds `HOSTNAME VARCHAR(64)` as the first column.
### Resolution
1. Replace old `*_TIME` SQL names with corresponding `*_DISTANCE` names.
2. Change SDK mappings from `DATETIME` to the original `DOUBLE`, `LONG`, or `ULONG`.
3. Read Cluster results by name or recheck ordinals including `HOSTNAME`.
4. Retain existing `*_TIME DATETIME` mappings for BASE TIME TAG queries.
See [Per-Tag Statistics Views](../query-analysis/#tag-stat-axis-schema) for the
complete mapping and Cluster aggregation precautions.
## Constraints and Precautions
### Feature Support
| Feature | Status |
|------|------|
| Time-series DATA UPDATE | Standard Edition; tag/BASETIME predicates required |
| Metadata UPDATE | Supported (`UPDATE ... METADATA`) |
| DELETE | Supported (`BEFORE`, tag/axis predicates, or full deletion) |
| Multiple PRIMARY KEY columns | Unsupported; one column only |
| BASETIME and BASEDISTANCE together | Unsupported |
| TAG DATA ordinary-column ALTER ADD/DROP | Unsupported |
| TAG METADATA ALTER ADD/DROP | Supported in Standard Edition |
### Tag Count Limits
- System settings limit the number of tags per TAG table.
- More tags increase tag-index and metadata memory use. Measure query and ingestion
performance with production-scale data.
- Do not make tag names unique per record (an antipattern; see
[One Table per Sensor](/dbms/data-modeling-table-design/table-types-patterns-type-anti/#per-sensor-create)).
### Late-Arriving Data
- BASETIME accepts arbitrary past timestamps.
- Measure ingestion rates and query performance separately for workloads with
substantial late-arriving data.
### Cluster Edition Support
Cluster Edition supports TAG tables.
However, TAG data UPDATE is Standard Edition only and is not supported in Cluster.
### Summary
```
TAG table = sensor name (PK) + time/distance axis + measurements
- INSERT/APPEND: O
- UPDATE: DATA requires Standard Edition tag/BASETIME predicates; METADATA uses separate SQL
- DELETE: O (BEFORE or tag/axis predicates)
- METADATA: O (separate attributes; UPDATE supported)
```
---
**Read Next**
- [TRANSACTION Table Design](/dbms/rdb-table-usage/)
## Clean Up the Exercise
Verify successful changes with SELECT, then remove only the exercise tables.
```sql
SELECT name, time, value, status FROM ch5_error_time ORDER BY name, time;
DROP TABLE ch5_error_time;
DROP TABLE ch5_error_distance;
```
---
title: "5.9 Usage Patterns and Scenarios"
url: https://docs.machbase.com/dbms/tag-table-usage/patterns-scenarios/
language: en
kind: page
---
# 5.9 Usage Patterns and Scenarios
Each example is independent. Check for conflicting objects before creating
tables. Define tag names, the meaning of one observation, value units, and
missing-data policy before applying DDL.
## Use Cases
### IoT Sensor Data
Manage sensor data from factories, buildings, and infrastructure in one TAG table.
```sql
CREATE TAG TABLE factory_sensor (
name VARCHAR(128) PRIMARY KEY,
time DATETIME BASETIME,
temperature DOUBLE,
vibration DOUBLE,
current DOUBLE
);
-- One row groups measurements from the same equipment observation.
INSERT INTO factory_sensor VALUES (
'F01/LINE-A/MOTOR-01',
NOW,
75.3, 0.15, 2.4
);
```
This model stores simultaneous measurements from one device in multiple
columns. For per-measurement tags such as `.../TEMP` and `.../VIBRATION`,
consider a single-value `name, time, value` model. Represent differences in
measurement times with NULL and quality status.
### Energy Monitoring
Collect electricity, gas, and water meter data over time.
```sql
CREATE TAG TABLE energy_meter (
meter_id VARCHAR(64) PRIMARY KEY,
time DATETIME BASETIME,
kwh DOUBLE,
voltage DOUBLE,
current DOUBLE
);
```
If `kwh` is cumulative, AVG(kwh) is the average meter reading, not interval
consumption. Calculate consumption from boundary differences with rules
for resets, replacements, and missing values. Distinguish power kW from
energy kWh and record units in metadata or the ingestion contract.
### Vehicle and Asset Tracking
Record GPS coordinates and speed as time series.
```sql
CREATE TAG TABLE vehicle_track (
vehicle_id VARCHAR(32) PRIMARY KEY,
time DATETIME BASETIME,
lat DOUBLE,
lon DOUBLE,
speed DOUBLE,
heading DOUBLE
);
```
Specify coordinate reference systems and latitude, longitude, and speed
units in the ingestion contract. Replacing missing positions with 0 makes
them indistinguishable from real coordinates. Creating this table does
not automatically provide spatial indexes or route matching. Query by
vehicle/time range first, then perform the required analysis.
### Unsuitable Cases
- A different tag name for every record (exploding tag counts)
- Frequent full-data UPDATE without tag/time limits
- Simple event logs (prefer LOG tables)
## Verify Results and Clean Up
For the IoT example, verify that the following query returns the three
measurements in one row.
```sql
SELECT name, time, temperature, vibration, current FROM factory_sensor;
DROP TABLE factory_sensor;
DROP TABLE energy_meter;
DROP TABLE vehicle_track;
```
Apply cleanup only to tables actually created on this page.
---
title: "5.10 TAG Metadata"
url: https://docs.machbase.com/dbms/tag-table-usage/tag-metadata/
language: en
kind: page
---
# 5.10 TAG Metadata
## Tag Metadata
### Overview
METADATA stores one row of current attributes per tag. Ordinary TAG queries repeat
those attributes with each DATA row. Changing a current attribute may also change
what historical DATA queries display. Preserve event-time attributes in DATA or a
separate attribute history when required.
The basic, JSON, and complete examples below use separate tables. Use metadata for
static tag attributes such as sensor location, equipment status, installation
details, external identifiers, and JSON documents.
Metadata-specific SQL supports the following operations. `TAG` in these examples
is a table name; replace it with your actual TAG table name.
- Query metadata only
- `UPDATE` / `DELETE` using metadata predicates
- Query the last modification time of a metadata row
- ADD/DROP ARRAY metadata columns and set DEFAULT for existing rows
- Declare `JSON` metadata columns
- Query and index JSON paths
- Update part of a JSON document
Use `TAG METADATA` syntax without accessing internal storage tables directly.
### Define Metadata Columns
Define metadata columns in the `METADATA (...)` clause of `CREATE TAG TABLE`.
```sql
CREATE TAG TABLE ch5_meta (
name VARCHAR(20) PRIMARY KEY,
time DATETIME BASETIME,
value DOUBLE
)
METADATA (
location VARCHAR(100),
status VARCHAR(20),
srcip IPV4
);
```
Metadata stores only one row per tag name.
### Add and Drop ARRAY Metadata Columns
Standard Edition can add and drop fixed-length numeric ARRAY columns in the METADATA
area of an existing TAG table.
```sql
INSERT INTO ch5_meta (name, time, value)
VALUES ('TEMP_OLD', TO_DATE('2026-09-05 00:00:00'), 10.0);
ALTER TABLE ch5_meta METADATA
ADD COLUMN (limits DECIMAL(12,4)[2] DEFAULT [0.0000, NULL]);
INSERT INTO ch5_meta (name, time, value)
VALUES ('TEMP_NEW', TO_DATE('2026-09-05 00:00:01'), 20.0);
SELECT name, limits
FROM ch5_meta METADATA
ORDER BY name;
```
The existing `TEMP_OLD` metadata row is backfilled with `[0.0000, NULL]`. The
`TEMP_NEW` row automatically registered by TAG DATA input after ALTER does not
reapply the ADD COLUMN DEFAULT; `limits` is whole NULL. Without a DEFAULT,
pre-ALTER rows also receive whole NULL.
Added ARRAY metadata columns appear in explicit projections and `SELECT *` in
ordinary TAG queries. ARRAY metadata columns have no automatic indexes and do
not support explicit indexes such as:
```sql
-- Unsupported; raises an error.
CREATE INDEX idx_sensor_limits ON ch5_meta METADATA(limits);
```
Specify `METADATA` when dropping a column as well.
```sql
ALTER TABLE ch5_meta METADATA DROP COLUMN (limits);
```
Ordinary ARRAY columns in TAG DATA can be declared in `CREATE TAG TABLE`, but
cannot be added with ALTER. See [Numeric ARRAY Types](/dbms/reference/sql/types/array/)
for element types, cardinality, and DEFAULT rules.
### Insert Metadata
Use `INSERT INTO ... METADATA` to insert metadata.
```sql
INSERT INTO ch5_meta METADATA VALUES (
'TEMP_001',
'Building-A/F1',
'READY',
'192.168.0.11'
);
```
You can also specify a column list.
```sql
INSERT INTO ch5_meta METADATA (name, status, srcip, location)
VALUES ('TEMP_002', 'STOP', '192.168.0.12', 'Building-A/F2');
```
Notes:
- Without a column list, VALUES follow the tag name and metadata declaration order.
- With a column list, values follow that list.
- NULL/DEFAULT handling for omitted input follows the DDL and input-path rules.
- The identifier is the TAG name column, declared as `name` in this example.
- Creating a metadata row automatically records server time in `_LAST_UPDATE_TIME`.
### Query Metadata
#### Query Metadata Only
Use `FROM TAG METADATA` for metadata-only queries.
```sql
SELECT name, location, status, srcip
FROM ch5_meta METADATA
ORDER BY name;
```
This returns one row per tag name.
```sql
SELECT *
FROM ch5_meta METADATA
ORDER BY name;
```
`SELECT *` and `table_alias.*` return only `NAME` and metadata columns.
System-managed columns such as `_LAST_UPDATE_TIME` do not appear in `SELECT *`.
Specify their names explicitly when needed.
#### Query the Last Modification Time
TAG metadata has a system-managed `_LAST_UPDATE_TIME` column recording each
metadata row's last modification time.
`_LAST_UPDATE_TIME` records metadata row creation or an actual metadata value
change, not the latest tag DATA insertion.
##### Query Methods
Query `_LAST_UPDATE_TIME` explicitly by name.
```sql
SELECT name, _last_update_time
FROM ch5_meta METADATA;
```
It can be selected with other metadata columns or used in predicates.
```sql
SELECT name, location, status, _last_update_time
FROM ch5_meta METADATA
WHERE name = 'TEMP_001';
```
`SELECT *` and `table_alias.*` omit `_LAST_UPDATE_TIME`.
##### Automatic Recording and Update Rules
Creating a metadata row records `_LAST_UPDATE_TIME` automatically.
```sql
INSERT INTO ch5_meta METADATA(name, location, status)
VALUES('TEMP_003', 'Building-A/F3', 'READY');
```
An actual user metadata value change updates `_LAST_UPDATE_TIME`.
```sql
UPDATE ch5_meta METADATA
SET status = 'DONE'
WHERE name = 'TEMP_003';
```
Updating to the same value, or removing a missing JSON path without changing
the stored result, is not an actual change. `_LAST_UPDATE_TIME` remains unchanged.
```sql
UPDATE ch5_meta METADATA
SET status = 'DONE'
WHERE name = 'TEMP_003';
```
Removing a nonexistent JSON path is also a no-op if the stored value does not
change. Run the example after creating the JSON table below.
##### Restrictions on Direct Writes
`_LAST_UPDATE_TIME` is system-managed. Users cannot insert or modify it directly.
The following statements are not allowed:
```sql
INSERT INTO ch5_meta METADATA(name, location, status, _last_update_time)
VALUES('TEMP_004', 'Building-A/F4', 'READY', now);
```
```sql
UPDATE ch5_meta METADATA
SET _last_update_time = now
WHERE name = 'TEMP_003';
```
`_LAST_UPDATE_TIME` is also prohibited as a TAG name column, TAG metadata column,
or target of `ALTER TABLE ... METADATA ADD COLUMN`. It cannot be removed with
`ALTER TABLE ... METADATA DROP COLUMN`.
```sql
CREATE TAG TABLE invalid_sensor (
_last_update_time VARCHAR(128) PRIMARY KEY,
time DATETIME BASETIME,
value DOUBLE
);
```
```sql
CREATE TAG TABLE invalid_sensor_meta (
name VARCHAR(128) PRIMARY KEY,
time DATETIME BASETIME,
value DOUBLE
)
METADATA (
_last_update_time DATETIME
);
```
Names sharing only the prefix, such as `_LAST_UPDATE_TIME2`, are allowed as user columns.
##### Time Predicates and Automatic Index
An index on `_LAST_UPDATE_TIME` is provided automatically for time predicates.
```sql
SELECT name, location, _last_update_time
FROM ch5_meta METADATA
WHERE _last_update_time >= TO_DATE('2026-06-08 00:00:00')
ORDER BY _last_update_time;
```
There is no need to create another index on the same column.
##### machloader / tagmetaimport Considerations
When importing TAG metadata, include only `NAME` and user metadata columns in
input/form files. Internal `_ID` and system-managed `_LAST_UPDATE_TIME` are not input targets.
For metadata columns `location` and `status`, use this input form:
```text
TEMP_001,Building-A/F1,READY
TEMP_002,Building-A/F2,STOP
```
The server fills `_LAST_UPDATE_TIME` automatically during import.
A user-defined `_LAST_UPDATE_TIME` in an ordinary LOG, LOOKUP, or VOLATILE table
behaves as an ordinary column. Reserved behavior applies only to the TAG metadata
system column.
#### Query with Data
Use ordinary `FROM TAG` to query time-series data using metadata predicates.
```sql
SELECT name, status, time, value
FROM ch5_meta
WHERE status = 'READY'
ORDER BY name, time;
```
Results are DATA-row based, so a tag's metadata values repeat with each DATA row.
Notes:
- `FROM TAG METADATA` cannot select DATA columns such as `TIME` or `VALUE`.
- `FROM TAG` selects DATA mode; `FROM TAG METADATA` selects metadata mode.
- Internal `_ID` and `_RID` columns are unavailable in `TAG METADATA`.
### Update Metadata
Use `UPDATE TAG METADATA` to change metadata.
```sql
UPDATE ch5_meta METADATA
SET status = 'DONE',
srcip = '10.0.0.20'
WHERE name = 'TEMP_001';
```
Metadata predicates can update multiple tags at once.
```sql
UPDATE ch5_meta METADATA
SET status = 'DONE'
WHERE status = 'READY';
```
Notes:
- Update targets are `NAME` and metadata columns.
- `UPDATE ... METADATA` cannot change DATA columns such as `TIME` or `VALUE`.
- Internal columns cannot be changed.
- `_LAST_UPDATE_TIME` changes only when metadata values actually change.
### Delete Metadata
Use `DELETE FROM TAG METADATA` to delete metadata.
To delete one tag's metadata, specify its name in `WHERE`.
```sql
DELETE FROM ch5_meta METADATA
WHERE name = 'TEMP_002';
```
Metadata predicates can delete multiple tags at once.
```sql
DELETE FROM ch5_meta METADATA
WHERE status = 'STOP';
```
Without `WHERE`, all metadata is targeted. The exercise still contains DATA for
TEMP_OLD and TEMP_NEW, so the following full deletion intentionally fails.
```sql
DELETE FROM ch5_meta METADATA;
```
Notes:
- If any target has DATA rows, the entire statement fails.
- Metadata for a tag in use cannot be deleted.
- Full deletion also fails entirely if any tag is in use; it does not delete only unused tags.
To delete metadata for a tag in use, delete its DATA rows first, then retry the
metadata deletion.
```sql
DELETE FROM ch5_meta
WHERE name = 'TEMP_001';
DELETE FROM ch5_meta METADATA
WHERE name = 'TEMP_001';
```
### JSON Metadata Columns
Metadata can contain `JSON` columns.
```sql
CREATE TAG TABLE ch5_meta_json (
name VARCHAR(20) PRIMARY KEY,
time DATETIME BASETIME,
value DOUBLE
)
METADATA (
status VARCHAR(20),
info JSON
);
```
Example JSON metadata input:
```sql
INSERT INTO ch5_meta_json METADATA VALUES (
'SHIP_001',
'READY',
'{"name":"alpha","ship":{"status":"READY"}}'
);
```
Notes:
- Do not specify a length for `JSON` metadata columns.
- Invalid JSON strings raise errors.
- The raw JSON column itself is not indexed automatically.
### When a JSON Value Does Not Change
```sql
SELECT name, _last_update_time FROM ch5_meta_json METADATA;
UPDATE ch5_meta_json METADATA
SET info = JSON_REMOVE(info, '$.missing')
WHERE name = 'SHIP_001';
SELECT name, _last_update_time FROM ch5_meta_json METADATA;
```
If a missing path leaves the stored value unchanged, the modification time also remains unchanged.
### Query JSON Paths
Query JSON metadata with the `->` operator.
```sql
SELECT name,
info->'$.name',
info->'$.ship.status'
FROM ch5_meta_json METADATA
WHERE info->'$.ship.status' = 'READY'
ORDER BY name;
```
Use the same syntax in DATA queries.
```sql
SELECT name, time, value
FROM ch5_meta_json
WHERE info->'$.ship.status' = 'READY'
ORDER BY name, time;
```
#### Path Notation
Queries and partial updates use full JSONPath syntax.
- Ordinary key: `$.name`
- Nested key: `$.ship.status`
- Use bracket notation for keys containing `.` or `-`.
```sql
SELECT info->'$[''ship.owner'']'
FROM ch5_meta_json METADATA;
SELECT info->'$[''ship-owner'']'
FROM ch5_meta_json METADATA;
```
### JSON Path Indexes
#### Declare Indexes at Table Creation
Index frequently queried JSON paths in the metadata definition.
```sql
CREATE TAG TABLE ch5_meta_json_indexed (
name VARCHAR(20) PRIMARY KEY,
time DATETIME BASETIME,
value DOUBLE
)
METADATA (
status VARCHAR(20),
info JSON INDEX('name', 'ship.status')
);
```
Strings inside `INDEX(...)` are interpreted as follows:
- `'name'` means `$.name`.
- `'ship.status'` means `$.ship.status`.
- Use full JSONPath for special-character keys or complex paths.
```sql
INFO JSON INDEX('$[''ship.owner'']')
```
#### Add Indexes After Creation
JSON path indexes can also be added after table creation.
```sql
CREATE INDEX idx_ship_owner
ON ch5_meta_json METADATA (info->'$.owner');
```
#### Drop an Index
Drop an index by its name.
```sql
SHOW INDEX idx_ship_owner;
DROP INDEX idx_ship_owner;
```
Manage explicitly created indexes by their declared names. Run
`SHOW INDEX idx_ship_owner;` before DROP. After deletion, the same name refers
to a nonexistent object.
#### Index Considerations
Current JSON path indexes primarily support string comparisons.
```sql
SELECT name
FROM ch5_meta_json METADATA
WHERE info->'$.status' = 'READY';
```
String-literal comparisons can use indexes. Numeric-literal comparisons may use a full scan.
Examples:
- `info->'$.num' = '10'`: May use an index
- `info->'$.num' = 10`: May use a full scan
### Partial JSON Updates
JSON functions return a new document value with the specified path changed.
UPDATE stores that result in the column. This is a logical path-level update,
not a performance guarantee of modifying only part of a storage file in place.
#### JSON_SET
Stores an SQL scalar as a JSON scalar.
```sql
UPDATE ch5_meta_json METADATA
SET info = JSON_SET(info, '$.ship.status', 'DONE')
WHERE name = 'SHIP_001';
```
#### JSON_SET_JSON
Parses the input string as JSON and stores an object or array.
```sql
UPDATE ch5_meta_json METADATA
SET info = JSON_SET_JSON(info, '$.owner', '{"name":"machbase","team":"db"}')
WHERE name = 'SHIP_001';
```
#### JSON_REMOVE
Removes a member or nested path.
```sql
UPDATE ch5_meta_json METADATA
SET info = JSON_REMOVE(info, '$.owner.team')
WHERE name = 'SHIP_001';
```
#### Partial Update Rules
- `JSON_SET(..., path, NULL)` stores JSON `null`.
- `JSON_SET_JSON(..., path, NULL)` returns SQL `NULL`.
- A `NULL` JSON document argument returns SQL `NULL`.
- A `NULL` or empty path raises an error.
- `JSON_REMOVE` of a nonexistent path is a no-op, not an error.
- `JSON_REMOVE(..., '$')` is not allowed.
- Partial updates primarily support object paths.
- Array-element path updates, such as `$.items[0]`, are unsupported.
### Complete Example
```sql
CREATE TAG TABLE ch5_meta_complete (
name VARCHAR(20) PRIMARY KEY,
time DATETIME BASETIME,
value DOUBLE
)
METADATA (
status VARCHAR(20),
srcip IPV4,
info JSON INDEX('name', 'ship.status')
);
INSERT INTO ch5_meta_complete METADATA VALUES (
'SHIP_001',
'READY',
'192.168.0.11',
'{"name":"alpha","ship":{"status":"READY"}}'
);
INSERT INTO ch5_meta_complete VALUES ('SHIP_001', '2026-04-01 00:00:00', 10.5);
SELECT name, status, info
FROM ch5_meta_complete METADATA;
SELECT name, time, value
FROM ch5_meta_complete
WHERE info->'$.ship.status' = 'READY';
CREATE INDEX idx_ship_owner
ON ch5_meta_complete METADATA (info->'$.owner');
UPDATE ch5_meta_complete METADATA
SET info = JSON_SET(info, '$.ship.status', 'DONE')
WHERE name = 'SHIP_001';
DROP INDEX idx_ship_owner;
```
### Summary
- Metadata-only queries: `FROM TAG METADATA`
- DATA queries: `FROM TAG`
- Metadata updates/deletions: `UPDATE/DELETE ... METADATA`
- ARRAY metadata changes: `ALTER TABLE ... METADATA ADD/DROP COLUMN`
- JSON metadata: `INFO JSON`
- `_LAST_UPDATE_TIME` is the metadata row's last modification time and can be queried explicitly.
- JSON path indexes: `INFO JSON INDEX(...)` or `CREATE INDEX ... ON TAG METADATA (...)`
- Partial JSON updates: `JSON_SET`, `JSON_SET_JSON`, `JSON_REMOVE`
- The server manages `_LAST_UPDATE_TIME` identically in Standard and Cluster environments.
## Clean Up the Exercise
Unlike the full-deletion failure example, DROP removes the table, DATA, and METADATA.
Verify that these names belong to objects created for this exercise before executing.
```sql
DROP TABLE ch5_meta;
DROP TABLE ch5_meta_json;
DROP TABLE ch5_meta_json_indexed;
DROP TABLE ch5_meta_complete;
```
---
title: "5.11 TAG Data UPDATE and Correction"
url: https://docs.machbase.com/dbms/tag-table-usage/tag-data-update-correction/
language: en
kind: page
---
# 5.11 TAG Data UPDATE and Correction
TAG DATA correction can directly replace original values or preserve originals
and apply corrected values at query time. These approaches affect ROLLUP
differently. UPDATE exercises require Machbase DBMS 8.7.0 Standard Edition.
## Correct Values Directly
Specify both tag-name and BASETIME predicates. SET expressions cannot reference
existing-row columns; pass a precomputed value or bind parameter instead of
`SET value = value + 1`. Create the following table with a nonconflicting name.
It includes default ROLLUP to verify aggregate correction.
```sql
CREATE TAG TABLE ch5_correction (
name VARCHAR(64) PRIMARY KEY,
time DATETIME BASETIME,
value DOUBLE SUMMARIZED,
status INTEGER
) WITH ROLLUP;
INSERT INTO ch5_correction VALUES
('TEMP-01', TO_DATE('2026-01-01 12:00:00', 'YYYY-MM-DD HH24:MI:SS'), 10.0, 0);
INSERT INTO ch5_correction VALUES
('TEMP-01', TO_DATE('2026-01-01 12:30:00', 'YYYY-MM-DD HH24:MI:SS'), 99.0, 0);
INSERT INTO ch5_correction VALUES
('TEMP-02', TO_DATE('2026-01-01 12:30:00', 'YYYY-MM-DD HH24:MI:SS'), 20.0, 0);
```
### Check Scope, Update, and Query Again
```sql
SELECT COUNT(*), MIN(value), MAX(value)
FROM ch5_correction
WHERE name = 'TEMP-01'
AND time >= TO_DATE('2026-01-01 12:00:00', 'YYYY-MM-DD HH24:MI:SS')
AND time < TO_DATE('2026-01-01 13:00:00', 'YYYY-MM-DD HH24:MI:SS');
UPDATE ch5_correction SET value = 25.0, status = 1
WHERE name = 'TEMP-01'
AND time >= TO_DATE('2026-01-01 12:00:00', 'YYYY-MM-DD HH24:MI:SS')
AND time < TO_DATE('2026-01-01 13:00:00', 'YYYY-MM-DD HH24:MI:SS');
SELECT name, time, value, status
FROM ch5_correction ORDER BY name, time;
```
The first query returns 2 rows, minimum 10.0, and maximum 99.0. After UPDATE,
both TEMP-01 rows have value=25.0 and status=1; TEMP-02 retains 20.0. A preceding
COUNT does not lock targets. Control the cutoff and range separately when
concurrent ingestion is possible.
### Multiple Tags and Error Handling
Tag selection supports `=`, `IN`, and `LIKE`. Check target names and counts
before broad patterns or long time ranges, and divide work into smaller ranges.
Multiple tags may be processed sequentially; do not assume a failure atomically
undoes the entire statement. Requery changed values and remaining targets before retrying.
Use `>= start AND < end` for adjacent time intervals to avoid processing boundary
rows twice. Ending a full day at `23:59:59` can miss fractional-second data
after that time. See [TAG UPDATE](../../reference/sql/syntax/dml-syntax/tag-data-update-syntax/)
for allowed predicates and binding.
### Rebuild ROLLUP
Changing source data does not automatically update previously calculated ROLLUP.
Rebuild the corrected interval as follows, then follow
[ROLLUP Rebuild](../../tag-rollup-usage/rollup-rebuild/) for progress and query validation.
```sql
EXEC ROLLUP_REBUILD(ch5_correction, 'TEMP-01',
TO_DATE('2026-01-01 12:00:00', 'YYYY-MM-DD HH24:MI:SS'),
TO_DATE('2026-01-01 13:00:00', 'YYYY-MM-DD HH24:MI:SS'));
```
## Preserve Originals and Correct to NULL
Store original and corrected values in separate columns to retain the initial
value. A corrected value may itself be NULL, so use a flag instead of testing
only `corrected_value IS NOT NULL`.
```sql
CREATE TAG TABLE ch5_correction_overlay (
name VARCHAR(64) PRIMARY KEY,
time DATETIME BASETIME,
raw_value DOUBLE,
corrected_value DOUBLE,
is_corrected SHORT
);
INSERT INTO ch5_correction_overlay VALUES
('TEMP-01', TO_DATE('2026-01-01 12:00:00', 'YYYY-MM-DD HH24:MI:SS'), 99.0, NULL, 0);
UPDATE ch5_correction_overlay
SET corrected_value = NULL, is_corrected = 1
WHERE name = 'TEMP-01'
AND time = TO_DATE('2026-01-01 12:00:00', 'YYYY-MM-DD HH24:MI:SS');
SELECT name, raw_value, is_corrected,
CASE WHEN is_corrected = 1 THEN corrected_value ELSE raw_value END AS effective_value
FROM ch5_correction_overlay;
```
raw_value is 99.0, is_corrected is 1, and effective_value is NULL. A flag of 0
selects the original. This approach chooses values in a query expression without
changing raw_value. Do not assume default ROLLUP aggregates the CASE expression
automatically or that rebuilding raw_value incorporates corrections. Define a
separate query/aggregation model for effective values.
## Correction History and Validation
Record separate history to retain reasons and operators for repeated changes.
```sql
CREATE LOG TABLE ch5_correction_log (
sensor_name VARCHAR(64),
target_time DATETIME,
old_value DOUBLE,
new_value DOUBLE,
reason VARCHAR(256),
corrected_by VARCHAR(64)
);
```
TAG updates and LOG history writes do not share one TRANSACTION transaction.
Design ordering, partial-failure handling, and operation identifiers for retries
in the application. Creating a history table does not automatically record audits.
After correction, verify original/effective values, affected-row counts, interval
statistics, and reports. Then remove only exercise objects. The CASCADE below
also deletes the ROLLUPs of ch5_correction.
```sql
DROP TABLE ch5_correction CASCADE;
DROP TABLE ch5_correction_overlay;
DROP TABLE ch5_correction_log;
```
---
title: "5.12 tagmetaimport and Bulk Metadata Registration"
url: https://docs.machbase.com/dbms/tag-table-usage/tagmetaimport/
language: en
kind: page
---
# 5.12 tagmetaimport and Bulk Metadata Registration
## Register Metadata with tagmetaimport
`tagmetaimport` imports tag names and user metadata from CSV. Distinguish
the logical TAG name used in SQL from the `-t` input target. The current
wrapper passes `-t` to machloader. For logical table `ch5_meta_import`,
the metadata input target is `_CH5_META_IMPORT_META`. Do not assume
`-t ch5_meta_import` automatically selects METADATA.
Use that name only to specify the tool target. Use `ch5_meta_import METADATA`
for SQL queries/changes; do not extend this into direct modification of
internal storage objects. Specify `-t` instead of relying on a default.
## 1. Prepare the Table
Run this SQL in an exercise database without conflicting objects.
```sql
CREATE TAG TABLE ch5_meta_import (
name VARCHAR(40) PRIMARY KEY,
time DATETIME BASETIME,
value DOUBLE
) METADATA (
location VARCHAR(40),
status VARCHAR(20)
);
```
## 2. Prepare CSV
Save the following as `ch5_metadata.csv` on the client.
```csv
name,location,status
TEMP_001,Building-A/F1,READY
TEMP_002,Building-A/F2,STOP
TEMP_003,Building-B/F3,READY
```
After the tag name, list values in METADATA declaration order. Omit DATA
time/value and system columns `_ID`/`_LAST_UPDATE_TIME`. Use `-H` for a
header; do not assume headers automatically map arbitrary column orders.
## 3. Import and Verify
Adjust the address/account to the exercise server and run with the
`MACHBASE_HOME` and library environment of the installed 8.7.0 package.
```bash
tagmetaimport -s 127.0.0.1 -P 5656 -u SYS -p MANAGER \
-t _CH5_META_IMPORT_META -d ch5_metadata.csv -H \
-l ch5_import.log -b ch5_import.bad
```
Expect 3 successes and 0 failures on the first run. The METADATA query
below returns three rows and DATA COUNT is 0. Registering metadata is
separate from inserting measurements.
```sql
SELECT name, location, status, _last_update_time
FROM ch5_meta_import METADATA ORDER BY name;
SELECT COUNT(*) FROM ch5_meta_import;
```
## 4. Existing Tags and Reruns
Reimporting the same file does not automatically update existing tags.
This path uses ordinary METADATA INSERT, so duplicate tags count as failed
rows. Expect 0 successes and 3 failures on the second run, with existing
attributes unchanged. Check success/failure counts and bad/log files,
not just exit status.
Do not assume a file containing valid new rows and invalid rows forms one
transaction. Check registered tags, correct failed rows, and reprocess only
those rows. Change existing attributes with explicit UPDATE or supported UPSERT.
```sql
UPDATE ch5_meta_import METADATA SET status = 'DONE' WHERE name = 'TEMP_001';
INSERT INTO ch5_meta_import METADATA VALUES ('TEMP_002', 'Building-C/F2', 'READY')
ON DUPLICATE KEY UPDATE;
SELECT name, location, status FROM ch5_meta_import METADATA ORDER BY name;
```
TEMP_001 changes to DONE; TEMP_002 changes to Building-C/F2 and READY.
Actual value changes update modification time; same-value no-ops preserve
it. Do not interpret this as an automatic UPSERT option in `tagmetaimport`.
## Cleanup and Related Documentation
After checking results, run `DROP TABLE ch5_meta_import;` to remove only
the exercise table. Remove CSV, log, and bad files after confirming they
are no longer needed for reprocessing.
See the [tagmetaimport Reference](../../reference/command-line-tools/tagmetaimport/)
for options and [TAG Metadata](../tag-metadata/) for SQL registration and change rules.
---
title: "6. ROLLUP for TAG Tables"
url: https://docs.machbase.com/dbms/tag-rollup-usage/
language: en
kind: section
---
# 6. ROLLUP for TAG Tables
ROLLUP preaggregates data from time-axis TAG tables and combines the stored statistics at query time
to reduce repeated analysis cost. This chapter distinguishes basic, conditional, extension, JSON,
and Custom ROLLUP in Machbase DBMS 8.7.0, covering creation, result validation, and rebuilding.
## Distinguish Three Intervals
| Concept | Meaning | Example |
|---|---|---|
| Creation INTERVAL | Width of stored aggregation buckets | 1 MIN |
| WAKEUP INTERVAL | How often the aggregation job wakes up | 10 SEC |
| Query bucket | Result interval requested by a report | `rollup('min', 5, time)` |
Multiple partial aggregates can be stored for the same bucket. Basic ROLLUP public query syntax
merges the required statistics; for Custom target TAG tables, users write the final reaggregation
query. Define source retention separately from ROLLUP retention and rebuild policies.
## Chapter Contents
| Section | Topics |
|---|---|
| [Overview and Use Criteria](./overview-use-criteria/) | Basic exercise and source/aggregate comparison |
| [Target TAG Design](./target-tag-table-design/) | ON/FROM, hierarchy constraints, and capacity estimates |
| [Creation and Deletion](./create-delete-rollup/) | CREATE, WITH ROLLUP, IF NOT EXISTS, and dependencies |
| [Query Syntax](./query-syntax-rollup/) | Candidate selection, time units, and origin |
| [Conditional ROLLUP](./conditional-rollup/) | Source filters and explicit candidate selection |
| [Custom ROLLUP](./custom-rollup/) | Reaggregating incremental results and OHLCV hierarchies |
| [Extension ROLLUP](./extension-rollup/) | FIRST/LAST and OHLC validation |
| [JSON ROLLUP](./json-summarized-rollup/) | Path/full-document aggregation and NULL |
| [Control and Status](./ingestion-control-rollup/) | STOP/START/WAKEUP/FORCE, V$ROLLUP, and gaps |
| [REBUILD](./rollup-rebuild/) | Supported targets, bucket boundaries, and corrections |
| [Performance Tuning](./performance-tuning-rollup/) | Comparing costs for equivalent results |
| [Usage Scenarios](./patterns-scenarios/) | Multiple tags and source/aggregate responsibilities |
Each page is a standalone exercise with object names beginning with `ch6_`. Check for conflicts with
existing business objects and keep intentional-error examples separate from success scripts. Query
fixed-time data using the stated fixed ranges. Clean up only the exercise tables and ROLLUPs.
Custom and REBUILD are Standard Edition only. Creating a ROLLUP or successfully ingesting data does
not mean aggregation has finished. Exercises use a named FORCE call to catch up processing and then
check results.
Continue with [Support Scope](../reference/support-scope-constraints/rollup/) and
[Troubleshooting](../troubleshooting/rollup/) for limitations and diagnosis.
---
title: "6.1 ROLLUP Overview and Use Criteria"
url: https://docs.machbase.com/dbms/tag-rollup-usage/overview-use-criteria/
language: en
kind: page
---
# 6.1 ROLLUP Overview and Use Criteria
ROLLUP reduces the cost of repeatedly aggregating raw rows. It is neither a source-retention policy
nor a cache for arbitrary query results, and it cannot reconstruct source information absent from
the aggregates.
## Use Criteria
| Requirement | Approach to consider |
|---|---|
| Repeated interval statistics for one numeric column | General ROLLUP |
| Aggregate only samples meeting quality conditions | Conditional ROLLUP |
| First and last values within an interval | EXTENSION ROLLUP |
| Store multiple aggregate expressions in a separate TAG | Custom ROLLUP (Standard Edition only) |
| Aggregate numeric values in JSON paths or documents | JSON path or full-document ROLLUP |
| Distance-axis TAG | Ordinary numeric interval aggregation; ROLLUP unsupported |
SUMMARIZED is not required when explicitly creating a ROLLUP on an ordinary numeric column. WITH
ROLLUP automatic creation and full-document JSON aggregation have separate SUMMARIZED requirements.
See [Creation Syntax](../create-delete-rollup/).
## Basic Exercise
### 1. Create and Insert
```sql
CREATE TAG TABLE ch6_basic (
name VARCHAR(32) PRIMARY KEY,
time DATETIME BASETIME,
value DOUBLE,
quality INTEGER
);
CREATE ROLLUP ch6_basic_ru ON ch6_basic(value) INTERVAL 1 MIN;
INSERT INTO ch6_basic VALUES ('TEMP_01', TO_DATE('2026-01-01 00:00:00', 'YYYY-MM-DD HH24:MI:SS'), 10.0, 1);
INSERT INTO ch6_basic VALUES ('TEMP_01', TO_DATE('2026-01-01 00:00:30', 'YYYY-MM-DD HH24:MI:SS'), 20.0, 1);
INSERT INTO ch6_basic VALUES ('TEMP_01', TO_DATE('2026-01-01 00:01:00', 'YYYY-MM-DD HH24:MI:SS'), 30.0, 1);
INSERT INTO ch6_basic VALUES ('TEMP_02', TO_DATE('2026-01-01 00:00:00', 'YYYY-MM-DD HH24:MI:SS'), 100.0, 1);
```
### 2. Check the Completed Aggregation Range
```sql
EXEC TABLE_FLUSH(ch6_basic);
ALTER ROLLUP ch6_basic_ru FORCE;
SHOW ROLLUPGAP;
```
SHOW ROLLUPGAP is a machsql command. SDKs should use supported SQL queries such as V$ROLLUP.
TABLE_FLUSH processes storage buffers; FORCE catches up the named ROLLUP's processing range. It does
not complete processing for future arrivals during continuous ingestion.
### 3. Compare with Source Data
```sql
SELECT DATE_TRUNC('minute', time) AS bucket,
COUNT(value), MIN(value), MAX(value), AVG(value)
FROM ch6_basic
WHERE name = 'TEMP_01'
GROUP BY bucket ORDER BY bucket;
SELECT rollup('min', 1, time) AS bucket,
COUNT(value), MIN(value), MAX(value), AVG(value)
FROM ch6_basic
WHERE name = 'TEMP_01'
GROUP BY bucket ORDER BY bucket;
```
| Bucket | COUNT(value) | MIN | MAX | AVG |
|---|---:|---:|---:|---:|
| 2026-01-01 00:00:00 | 2 | 10 | 20 | 15 |
| 2026-01-01 00:01:00 | 1 | 30 | 30 | 30 |
Both queries should return the same results. A DATE_TRUNC aggregation on source data does not
automatically switch simply because a ROLLUP exists. Specify `rollup()` explicitly in ROLLUP
queries.
### 4. Clean Up
```sql
DROP ROLLUP ch6_basic_ru;
DROP TABLE ch6_basic;
```
## Before Adoption
Define representative tag counts, ingestion volume, query frequency, and acceptable aggregation lag.
Determine the finest required interval and source retention first, then continue to
[Hierarchy Design](../target-tag-table-design/). Measure long-period performance with
production-like data rather than extrapolating from this small sample.
---
title: "6.2 Target TAG Table Design for ROLLUP"
url: https://docs.machbase.com/dbms/tag-rollup-usage/target-tag-table-design/
language: en
kind: page
---
# 6.2 Target TAG Table Design for ROLLUP
## ON and FROM
`ON source(column)` aggregates a column of a source time-axis TAG table. `FROM rollup_name` combines
statistics from an existing general/extension ROLLUP into larger intervals. A Custom target is also
a TAG table, but the next Custom stage uses INTO...AS with a SELECT from that target TAG.
## Hierarchy Exercise
```sql
CREATE TAG TABLE ch6_design (
name VARCHAR(32) PRIMARY KEY,
time DATETIME BASETIME,
value DOUBLE,
quality INTEGER
);
CREATE ROLLUP ch6_design_sec ON ch6_design(value) INTERVAL 1 SEC;
CREATE ROLLUP ch6_design_min FROM ch6_design_sec INTERVAL 1 MIN;
CREATE ROLLUP ch6_design_hour FROM ch6_design_min INTERVAL 1 HOUR;
INSERT INTO ch6_design VALUES ('TEMP_01', TO_DATE('2026-01-01 00:00:00', 'YYYY-MM-DD HH24:MI:SS'), 10.0, 1);
INSERT INTO ch6_design VALUES ('TEMP_01', TO_DATE('2026-01-01 00:00:30', 'YYYY-MM-DD HH24:MI:SS'), 20.0, 1);
INSERT INTO ch6_design VALUES ('TEMP_01', TO_DATE('2026-01-01 00:01:00', 'YYYY-MM-DD HH24:MI:SS'), 30.0, 1);
INSERT INTO ch6_design VALUES ('TEMP_02', TO_DATE('2026-01-01 00:00:00', 'YYYY-MM-DD HH24:MI:SS'), 100.0, 1);
EXEC TABLE_FLUSH(ch6_design);
ALTER ROLLUP ch6_design_sec FORCE;
ALTER ROLLUP ch6_design_min FORCE;
ALTER ROLLUP ch6_design_hour FORCE;
SELECT name, rollup('hour', 1, time) AS bucket,
SUM(value), COUNT(value), AVG(value)
FROM ch6_design
GROUP BY name, bucket ORDER BY name, bucket;
```
TEMP_01 has sum 60, count 3, and average 20; TEMP_02 has 100, 1, and 100. FORCE lower levels first
so upper levels can process newly generated lower-level results.
### Hierarchy Constraints
- The upper interval must be larger than the source interval and an integer multiple of it. Equal intervals are not allowed.
- FROM cannot convert general ROLLUP to extension ROLLUP or vice versa. Keep EXTENSION consistent throughout the hierarchy.
- JSON path/full-document modes must also match the source.
- Coarse statistics cannot reconstruct source detail finer than the smallest supported query interval.
Each creation example below intentionally fails.
```sql
CREATE ROLLUP ch6_design_bad_same FROM ch6_design_min INTERVAL 1 MIN;
CREATE ROLLUP ch6_design_bad_divisor FROM ch6_design_min INTERVAL 90 SEC;
CREATE ROLLUP ch6_design_bad_ext FROM ch6_design_sec INTERVAL 1 MIN EXTENSION;
```
## Choosing Intervals and Storage Capacity
Assuming every tag has values in every interval, the approximate logical bucket count is
`(retention time / bucket interval) × tag count`. Retaining 1-second buckets for 10,000 tags for 365
days gives about 315.4 billion buckets. Actual row counts depend on partial aggregates, empty
intervals, and column counts. Measure disk usage including compression and storage overhead.
If second-level observations are queried only by minute, assess whether every second-level hierarchy
is needed. Creation intervals use SEC/MIN/HOUR; DAY/WEEK/MONTH/YEAR are query bucket units. In
particular, do not assume a 24 HOUR storage interval works directly for daily queries; check
[Candidate Selection Rules](../query-syntax-rollup/).
A new ROLLUP initially aggregates existing data still present in its source. Check initial workload
and gaps. FORCE does not rewind processing after source corrections. Follow
[REBUILD Support Scope](../rollup-rebuild/).
## Clean Up
```sql
DROP ROLLUP ch6_design_hour;
DROP ROLLUP ch6_design_min;
DROP ROLLUP ch6_design_sec;
DROP TABLE ch6_design;
```
---
title: "6.3 Create and Delete ROLLUP"
url: https://docs.machbase.com/dbms/tag-rollup-usage/create-delete-rollup/
language: en
kind: page
---
# 6.3 Create and Delete ROLLUP
## Creation Syntax
```text
CREATE ROLLUP [IF NOT EXISTS] name
ON source_tag [(column_or_json_path)]
INTERVAL n (SEC|MIN|HOUR)
[WAKEUP INTERVAL m (SEC|MIN|HOUR)]
[EXTENSION]
[WHERE predicate];
CREATE ROLLUP [IF NOT EXISTS] name
FROM source_rollup
INTERVAL n (SEC|MIN|HOUR)
[WAKEUP INTERVAL m (SEC|MIN|HOUR)]
[EXTENSION]
[WHERE predicate];
```
Do not specify a separate extension name after EXTENSION. CREATE accepts SEC/MIN/HOUR, unlike
query-function units such as DAY/MONTH. The interval must be positive; the current validation
maximum is equivalent to 365 days. Source, hierarchy, and aggregation-mode requirements must also be
met.
| Target | Requirement |
|---|---|
| Ordinary numeric column | Specify a supported numeric type; SUMMARIZED is not required |
| JSON path | Specify the JSON column and a valid path |
| Entire JSON document | Requires a JSON SUMMARIZED column |
| WITH ROLLUP automatic creation | Requires the third column to be SUMMARIZED |
| METADATA, distance-axis, or non-TAG | Not a general ROLLUP target |
## Creation, Duplicate-Name Checks, and Queries
```sql
CREATE TAG TABLE ch6_create (
name VARCHAR(32) PRIMARY KEY,
time DATETIME BASETIME,
value DOUBLE,
quality INTEGER
);
CREATE ROLLUP IF NOT EXISTS ch6_create_ru ON ch6_create(value) INTERVAL 1 MIN;
CREATE ROLLUP IF NOT EXISTS ch6_create_ru ON ch6_create(value) INTERVAL 1 MIN;
INSERT INTO ch6_create VALUES ('TEMP_01', TO_DATE('2026-01-01 00:00:00', 'YYYY-MM-DD HH24:MI:SS'), 10.0, 1);
INSERT INTO ch6_create VALUES ('TEMP_01', TO_DATE('2026-01-01 00:00:30', 'YYYY-MM-DD HH24:MI:SS'), 20.0, 1);
INSERT INTO ch6_create VALUES ('TEMP_01', TO_DATE('2026-01-01 00:01:00', 'YYYY-MM-DD HH24:MI:SS'), 30.0, 1);
INSERT INTO ch6_create VALUES ('TEMP_02', TO_DATE('2026-01-01 00:00:00', 'YYYY-MM-DD HH24:MI:SS'), 100.0, 1);
EXEC TABLE_FLUSH(ch6_create);
ALTER ROLLUP ch6_create_ru FORCE;
SELECT DISTINCT ROLLUP_NAME, COLUMN_NAME, INTERVAL_TIME, WAKEUP_INTERVAL
FROM V$ROLLUP WHERE ROLLUP_NAME = 'CH6_CREATE_RU';
SELECT rollup('min', 1, time) AS bucket, AVG(value)
FROM ch6_create WHERE name = 'TEMP_01'
GROUP BY bucket ORDER BY bucket;
```
Recreating an existing name retains its definition. IF NOT EXISTS neither changes nor verifies the
definition, and does not bypass all invalid-SQL or source validation. Both interval columns are
60000 ms. The query averages are 15 at 00:00 and 30 at 00:01.
Creating the same name without IF NOT EXISTS causes an error. Check it separately from the
successful exercise.
```sql
CREATE ROLLUP ch6_create_ru ON ch6_create(value) INTERVAL 1 MIN;
```
## Automatic Creation with WITH ROLLUP
The following separate table automatically creates the default SEC→MIN→HOUR hierarchy.
```sql
CREATE TAG TABLE ch6_auto (
name VARCHAR(32) PRIMARY KEY,
time DATETIME BASETIME,
value DOUBLE SUMMARIZED
) WITH ROLLUP (SEC);
SELECT DISTINCT ROLLUP_NAME, ROOT_TABLE, INTERVAL_TIME, EXT_TYPE
FROM V$ROLLUP WHERE ROOT_TABLE = 'CH6_AUTO'
ORDER BY INTERVAL_TIME;
```
The query returns three rows with INTERVAL_TIME values 1000, 60000, and 3600000.
Automatic EXTENSION creation uses `WITH ROLLUP (SEC) EXTENSION`. Check actual generated names in
V$ROLLUP; do not assume name conflicts are resolved automatically.
The argument determines the hierarchy: `(SEC)` creates SEC, MIN, and HOUR; `(MIN)` creates MIN and
HOUR; `(HOUR)` creates only HOUR. Omitting the argument is equivalent to `(SEC)`. Only the first
stage reads the source TAG; each later stage reads the preceding ROLLUP. Only SEC/MIN/HOUR are
accepted. Query-function units such as DAY cause an error.
## Deleting and Changing Definitions
Remove upper-level dependents before a source referenced by another ROLLUP. A Custom target TAG
cannot be dropped before its job. To change a definition, account for readers and reaggregation
time, then switch to a new object or remove and recreate the existing definition.
```sql
DROP ROLLUP ch6_create_ru;
DROP TABLE ch6_create;
DROP TABLE ch6_auto CASCADE;
```
The final CASCADE also removes this exercise's automatic ROLLUPs. Do not use it as the default for
ordinary production cleanup. CASCADE on a Custom source can remove associated jobs, but does not
automatically delete user target TAG tables.
Conditional, extension, JSON, and Custom exercises are provided independently in their sections. For
full syntax, see the [SQL Reference](../../reference/sql/syntax/rollup-syntax/).
---
title: "6.4 ROLLUP Query Syntax"
url: https://docs.machbase.com/dbms/tag-rollup-usage/query-syntax-rollup/
language: en
kind: page
---
# 6.4 ROLLUP Query Syntax
## Explicit ROLLUP Queries
```text
rollup(time_unit, period, basetime_column [, origin])
```
| Argument | Requirements |
|---|---|
| time_unit | SECOND/SEC, MINUTE/MIN, HOUR, DAY, WEEK, MONTH, YEAR; case-insensitive |
| period | Positive integer literal; not a column or parameter placeholder |
| basetime_column | BASETIME column of the source time-axis TAG |
| origin | If omitted, uses the default reference point adjusted for the time-zone offset; explicit values have unit-specific restrictions |
The result is a DATETIME bucket. Finer detail than the minimum stored aggregation interval cannot be
reconstructed. If no applicable ROLLUP exists, the query fails instead of automatically falling back
to a raw scan. Write a separate DATE_TRUNC/DATE_BIN + GROUP BY query for source aggregation.
## Preparation and Basic Queries
```sql
CREATE TAG TABLE ch6_query (
name VARCHAR(32) PRIMARY KEY,
time DATETIME BASETIME,
value DOUBLE,
quality INTEGER
);
CREATE ROLLUP ch6_query_sec ON ch6_query(value) INTERVAL 1 SEC;
INSERT INTO ch6_query VALUES ('TEMP_01', TO_DATE('2026-01-01 00:00:00', 'YYYY-MM-DD HH24:MI:SS'), 10.0, 1);
INSERT INTO ch6_query VALUES ('TEMP_01', TO_DATE('2026-01-01 00:00:30', 'YYYY-MM-DD HH24:MI:SS'), 20.0, 1);
INSERT INTO ch6_query VALUES ('TEMP_01', TO_DATE('2026-01-01 00:01:00', 'YYYY-MM-DD HH24:MI:SS'), 30.0, 1);
INSERT INTO ch6_query VALUES ('TEMP_02', TO_DATE('2026-01-01 00:00:00', 'YYYY-MM-DD HH24:MI:SS'), 100.0, 1);
EXEC TABLE_FLUSH(ch6_query);
ALTER ROLLUP ch6_query_sec FORCE;
SELECT name, rollup('min', 1, time) AS bucket,
COUNT(value), SUM(value), MIN(value), MAX(value), AVG(value)
FROM ch6_query
WHERE time >= TO_DATE('2026-01-01 00:00:00', 'YYYY-MM-DD HH24:MI:SS')
AND time < TO_DATE('2026-01-01 00:02:00', 'YYYY-MM-DD HH24:MI:SS')
GROUP BY name, bucket ORDER BY name, bucket;
```
TEMP_01 at 00:00 has COUNT=2, SUM=30, AVG=15; at 00:01 it has 1, 30, 30. TEMP_02 at 00:00 has 1,
100, 100. If SELECT returns name, also include name in GROUP BY. Omitting name combines tags; do not
mix sensors with different units.
## Candidate Selection and Hints
1. If a ROLLUP_TABLE hint is present, check the named candidate for compatibility.
2. Automatic selection finds candidates matching the aggregate column, JSON path, mode, and requested interval.
3. Search unconditional candidates first; if none exist, include conditional candidates.
4. Select the largest applicable interval. For equal intervals, retain the first registered candidate.
Do not assume general ROLLUP always takes priority over extension ROLLUP. If only conditional
ROLLUPs exist, filtered data can be selected automatically. Check which sample set the result
represents. Specify a hint when a particular aggregate must be used.
```sql
SELECT /*+ ROLLUP_TABLE(ch6_query_sec) */
rollup('min', 1, time) AS bucket, AVG(value)
FROM ch6_query WHERE name = 'TEMP_01'
GROUP BY bucket ORDER BY bucket;
```
### Stored Candidate Intervals and Query Buckets
Current candidate-interval validation calculates SEC requests as period seconds and MIN requests as
period minutes. HOUR and DAY/WEEK/MONTH/YEAR requests are checked against period hours during
candidate selection. Selected statistics are then merged into the requested calendar/time buckets.
Do not assume a 24 HOUR ROLLUP automatically serves `rollup('day', 1, time)`. That request considers
HOUR/MIN/SEC candidates compatible with a 1-hour basis. Distinguish creation INTERVAL from result
buckets and inspect the actual execution plan.
## Aggregate Functions and Samples
General numeric ROLLUP supports MIN, MAX, SUM, COUNT, AVG, and SUMSQ. FIRST/LAST in a ROLLUP query
require EXTENSION. Custom results are stored in ordinary TAG tables; use the sum/count rules in
[Custom Reaggregation](../custom-rollup/). COUNT for full-document JSON aggregation is covered
separately in [JSON](../json-summarized-rollup/).
### SUMSQ, Variance, and Standard Deviation
ROLLUP queries do not directly support STDDEV, STDDEV_POP, VARIANCE, or VAR_POP. Using them in a
`rollup()` query causes
`ERR-02816: Only rollup column with aggregate function can be referenced in ROLLUP SELECT query.`
Variances and standard deviations cannot be added across intervals. Averaging two interval standard
deviations does not give the standard deviation of the combined interval.
Instead, ROLLUP stores SUMSQ, the sum of squared values, alongside COUNT and SUM. All three are
additive and remain valid when merged into buckets larger than the storage interval. Variance and
standard deviation can then be calculated at query time. SUMSQ is included in both general and
EXTENSION ROLLUP.
| Value | Formula |
|---|---|
| Population variance | `SUMSQ/N - (SUM/N)^2` |
| Population standard deviation | Square root of population variance |
| Sample variance | `(SUMSQ - SUM^2/N) / (N-1)` |
| Sample standard deviation | Square root of sample variance |
N is `COUNT(value)`, the number of valid, non-NULL values.
Keep only supported aggregates in the ROLLUP query block, and calculate variance and standard
deviation outside the inline view. This reads COUNT, SUM, and SUMSQ once and separates derived
calculations, allowing formulas to change without changing the ROLLUP query.
```sql
SELECT bucket, n, s, sq,
sq/n - POWER(s/n, 2) AS var_pop,
SQRT(sq/n - POWER(s/n, 2)) AS stddev_pop
FROM (
SELECT rollup('min', 1, time) AS bucket,
COUNT(value) AS n,
SUM(value) AS s,
SUMSQ(value) AS sq
FROM ch6_query WHERE name = 'TEMP_01'
GROUP BY bucket
) t
ORDER BY bucket;
```
The 00:00 bucket contains 10 and 20, so n=2, s=30, sq=500, population variance=25, and population
standard deviation=5. The 00:01 bucket has one value, so both population measures are 0.
Sample variance divides by `N-1`, so handle single-value buckets first, also outside the inline
view. Explicit `ELSE NULL` causes `ERR-02042`; omit ELSE.
```sql
SELECT bucket, n,
CASE WHEN n > 1
THEN (sq - POWER(s, 2)/n) / (n - 1)
END AS var_samp
FROM (
SELECT rollup('min', 1, time) AS bucket,
COUNT(value) AS n,
SUM(value) AS s,
SUMSQ(value) AS sq
FROM ch6_query WHERE name = 'TEMP_01'
GROUP BY bucket
) t
ORDER BY bucket;
```
The 00:00 sample variance is 50; the single-value 00:01 bucket returns NULL. To omit single-value
intervals, use `WHERE n > 1` outside the inline view. Source-table `VARIANCE` and `STDDEV` return 0
rather than NULL for these intervals, so align display policies when combining results.
For the example values, results match direct `VAR_POP`, `STDDEV_POP`, `VARIANCE`, and `STDDEV` on
the source table. However, both formulas lose precision when the mean is large relative to the
spread. For values near 100000 varying only in fractional digits, `SUMSQ/N` and `(SUM/N)^2` are
nearly equal, reducing significant digits after subtraction. If floating-point error yields a small
negative variance, SQRT is also invalid. When precision matters, compare with source-table STDDEV
and VAR_POP to establish an acceptable range.
## Calendar Units and origin
Use the same data to check monthly results and weeks starting on Monday.
```sql
SELECT rollup('month', 1, time, '2000-01-01 00:00:00') AS bucket,
SUM(value), COUNT(value), AVG(value)
FROM ch6_query WHERE name = 'TEMP_01'
GROUP BY bucket ORDER BY bucket;
SELECT rollup('week', 1, time, '1970-01-05 00:00:00') AS bucket, AVG(value)
FROM ch6_query WHERE name = 'TEMP_01'
GROUP BY bucket ORDER BY bucket;
```
The monthly query returns SUM=60, COUNT=3, AVG=20 in the 2026-01-01 bucket. Monthly/yearly origin
values must be on the first day of a month after time-zone interpretation, and results fall at
midnight on the corresponding month boundary. This does not shift monthly business start times by
arbitrary dates or time offsets. Distinguish fixed-interval origins, such as day/week, from
month/year calendar calculations.
Check string interpretation together with the connection time zone. Do not assume DST automatically
matches the desired business calendar. Compare against source DATE_BIN aggregation around
boundaries. Validate equivalence to source results for origins or query boundaries that would split
stored aggregates.
## Clean Up
```sql
DROP ROLLUP ch6_query_sec;
DROP TABLE ch6_query;
```
---
title: "6.5 Conditional ROLLUP"
url: https://docs.machbase.com/dbms/tag-rollup-usage/conditional-rollup/
language: en
kind: page
---
# 6.5 Conditional ROLLUP
## Filter Source Rows Before Aggregation
Conditional ROLLUP maintains statistics for source rows meeting quality or state conditions. This
differs from removing bad samples from an already computed average. The quality column used by the
predicate is not itself retained in the aggregate result.
## 1. Prepare the Table and Two ROLLUPs
```sql
CREATE TAG TABLE ch6_condition (
name VARCHAR(32) PRIMARY KEY,
time DATETIME BASETIME,
value DOUBLE,
quality INTEGER
);
CREATE ROLLUP ch6_condition_all
ON ch6_condition(value) INTERVAL 1 MIN EXTENSION;
CREATE ROLLUP ch6_condition_good
ON ch6_condition(value) INTERVAL 1 MIN EXTENSION WHERE quality = 1;
INSERT INTO ch6_condition VALUES ('TEMP_01', TO_DATE('2026-01-01 00:00:00', 'YYYY-MM-DD HH24:MI:SS'), 10.0, 1);
INSERT INTO ch6_condition VALUES ('TEMP_01', TO_DATE('2026-01-01 00:00:30', 'YYYY-MM-DD HH24:MI:SS'), 20.0, 1);
INSERT INTO ch6_condition VALUES ('TEMP_01', TO_DATE('2026-01-01 00:01:00', 'YYYY-MM-DD HH24:MI:SS'), 30.0, 1);
INSERT INTO ch6_condition VALUES ('TEMP_02', TO_DATE('2026-01-01 00:00:00', 'YYYY-MM-DD HH24:MI:SS'), 100.0, 1);
INSERT INTO ch6_condition VALUES
('TEMP_01', TO_DATE('2026-01-01 00:00:50', 'YYYY-MM-DD HH24:MI:SS'), 90.0, 0);
EXEC TABLE_FLUSH(ch6_condition);
ALTER ROLLUP ch6_condition_all FORCE;
ALTER ROLLUP ch6_condition_good FORCE;
```
## 2. Compare the Source Predicate and ROLLUP
```sql
SELECT DATE_TRUNC('minute', time) AS bucket, COUNT(value), AVG(value),
MIN(value), MAX(value), FIRST(time, value), LAST(time, value)
FROM ch6_condition
WHERE name = 'TEMP_01' AND quality = 1
GROUP BY bucket ORDER BY bucket;
SELECT /*+ ROLLUP_TABLE(ch6_condition_good) */
rollup('min', 1, time) AS bucket, COUNT(value), AVG(value),
MIN(value), MAX(value), FIRST(time, value), LAST(time, value)
FROM ch6_condition WHERE name = 'TEMP_01'
GROUP BY bucket ORDER BY bucket;
SELECT /*+ ROLLUP_TABLE(ch6_condition_all) */
rollup('min', 1, time) AS bucket, COUNT(value), AVG(value),
MIN(value), MAX(value), FIRST(time, value), LAST(time, value)
FROM ch6_condition WHERE name = 'TEMP_01'
GROUP BY bucket ORDER BY bucket;
```
| Bucket and set | COUNT | AVG | MIN | MAX | FIRST | LAST |
|---|---:|---:|---:|---:|---:|---:|
| 00:00, all rows | 3 | 40 | 10 | 90 | 10 | 90 |
| 00:00, quality=1 | 2 | 15 | 10 | 20 | 10 | 20 |
| 00:01, both sets | 1 | 30 | 30 | 30 | 30 | 30 |
The bad sample is last in its interval, so LAST also differs. FIRST/LAST return the selected value,
not a timestamp/value pair.
## Explicit Candidate Selection
This example uses hints to fix the result sets for all samples and valid samples. Automatic
selection favors unconditional candidates, but can select filtered statistics when only conditional
candidates exist. Do not interpret this as conditional ROLLUP always being ignored or EXTENSION
always requiring a hint.
## Syntax and Constraints
For general ROLLUP, place the filter in WHERE after INTERVAL and EXTENSION. Comparisons, BETWEEN,
IN, LIKE, logical operations, and supported scalar functions are allowed. Subqueries, aggregate
functions, and predicates on the tag-name PRIMARY KEY are unsupported. Custom uses WHERE inside
SELECT; do not mix these syntax forms.
## Check Status and Clean Up
```sql
SELECT DISTINCT ROLLUP_NAME, PREDICATE, ENABLED
FROM V$ROLLUP WHERE ROOT_TABLE = 'CH6_CONDITION';
DROP ROLLUP ch6_condition_good;
DROP ROLLUP ch6_condition_all;
DROP TABLE ch6_condition;
```
When business quality criteria change, update both the predicate and reaggregation plan. Existing
aggregates do not automatically adopt the new predicate. See [Control](../ingestion-control-rollup/)
and [Rebuild Scope](../rollup-rebuild/).
---
title: "6.6 Custom ROLLUP"
url: https://docs.machbase.com/dbms/tag-rollup-usage/custom-rollup/
language: en
kind: page
---
# 6.6 Custom ROLLUP
Standard Edition only
## Custom and General ROLLUP
Custom ROLLUP appends incremental SELECT aggregation results to a precreated target TAG table.
General ROLLUP reads internal statistics through `rollup()`; Custom instead requires querying the
target TAG directly and merging partial aggregates again. It cannot be created in Cluster Edition.
```text
CREATE ROLLUP [IF NOT EXISTS] name
INTO (destination_tag)
AS (SELECT ... FROM source_tag [WHERE ...] GROUP BY ...)
INTERVAL n (SEC|MIN|HOUR)
[WAKEUP INTERVAL m (SEC|MIN|HOUR)];
```
The source must be one time-axis TAG table. JOIN and FROM subqueries are unsupported. The target
must be a precreated TAG compatible with the SELECT column order and types. WHERE inside SELECT is
allowed, but direct BASETIME predicates are not. Do not append an external WHERE after INTERVAL as
in general ROLLUP.
Match the creation interval to the time bucket calculated by SELECT. Because the job processes only
new input, one bucket can contain several result rows. Do not interpret row count as bucket count.
## 1. Sum and Valid-Count Exercise
```sql
CREATE TAG TABLE ch6_custom_src (
name VARCHAR(32) PRIMARY KEY, time DATETIME BASETIME, value DOUBLE
);
CREATE TAG TABLE ch6_custom_dst (
name VARCHAR(32) PRIMARY KEY, time DATETIME BASETIME,
sum_value DOUBLE, valid_count LONG, total_count LONG
);
CREATE ROLLUP ch6_custom_ru INTO (ch6_custom_dst)
AS (
SELECT name, DATE_TRUNC('minute', time) AS time,
SUM(value), COUNT(value), COUNT(*)
FROM ch6_custom_src
GROUP BY name, time
) INTERVAL 1 MIN;
INSERT INTO ch6_custom_src VALUES ('S1', TO_DATE('2026-01-01 00:00:00', 'YYYY-MM-DD HH24:MI:SS'), 10);
INSERT INTO ch6_custom_src VALUES ('S1', TO_DATE('2026-01-01 00:00:05', 'YYYY-MM-DD HH24:MI:SS'), 10);
EXEC TABLE_FLUSH(ch6_custom_src);
ALTER ROLLUP ch6_custom_ru FORCE;
INSERT INTO ch6_custom_src VALUES ('S1', TO_DATE('2026-01-01 00:00:10', 'YYYY-MM-DD HH24:MI:SS'), 20);
INSERT INTO ch6_custom_src VALUES ('S1', TO_DATE('2026-01-01 00:00:15', 'YYYY-MM-DD HH24:MI:SS'), 20);
INSERT INTO ch6_custom_src VALUES ('S1', TO_DATE('2026-01-01 00:00:20', 'YYYY-MM-DD HH24:MI:SS'), 20);
INSERT INTO ch6_custom_src VALUES ('S1', TO_DATE('2026-01-01 00:00:25', 'YYYY-MM-DD HH24:MI:SS'), 20);
INSERT INTO ch6_custom_src VALUES ('S1', TO_DATE('2026-01-01 00:00:30', 'YYYY-MM-DD HH24:MI:SS'), 20);
INSERT INTO ch6_custom_src VALUES ('S1', TO_DATE('2026-01-01 00:00:35', 'YYYY-MM-DD HH24:MI:SS'), 20);
INSERT INTO ch6_custom_src VALUES ('S1', TO_DATE('2026-01-01 00:00:40', 'YYYY-MM-DD HH24:MI:SS'), 20);
INSERT INTO ch6_custom_src VALUES ('S1', TO_DATE('2026-01-01 00:00:45', 'YYYY-MM-DD HH24:MI:SS'), 20);
INSERT INTO ch6_custom_src VALUES ('S1', TO_DATE('2026-01-01 00:00:55', 'YYYY-MM-DD HH24:MI:SS'), NULL);
EXEC TABLE_FLUSH(ch6_custom_src);
ALTER ROLLUP ch6_custom_ru FORCE;
SELECT name, DATE_TRUNC('minute', time) AS bucket,
SUM(value), COUNT(value), COUNT(*), AVG(value)
FROM ch6_custom_src
GROUP BY name, bucket ORDER BY name, bucket;
SELECT name, time, SUM(sum_value) AS sum_value,
SUM(valid_count) AS valid_count, SUM(total_count) AS total_count,
CASE WHEN SUM(valid_count) = 0 THEN NULL
ELSE SUM(sum_value) / SUM(valid_count) END AS avg_value
FROM ch6_custom_dst
GROUP BY name, time ORDER BY name, time;
```
The bucket has sum 180, 10 valid values, 11 total rows, and average 18. This differs from 15, the
simple average of partial averages 10 and 20. Use COUNT(value) so NULL is excluded from the average
denominator, and retain COUNT(*) separately for total rows including NULL. Define a result policy
that avoids division by zero for all-NULL buckets.
### Sample Ratio and Time-Based Availability
Matching sample count divided by total sample count is a sample ratio. Interpreting it as time-based
availability requires accounting for observation intervals, missing data, and state duration. Sum
the numerator and denominator separately; do not average partial ratios.
## 2. OHLCV and a 1-Minute → 10-Minute Custom Hierarchy
This is a separate exercise. Store the first and last source observation timestamps to support
FIRST/LAST reaggregation.
```sql
CREATE TAG TABLE ch6_ticks (
code VARCHAR(20) PRIMARY KEY, time DATETIME BASETIME,
price DOUBLE, volume DOUBLE
);
CREATE TAG TABLE ch6_candle_min (
code VARCHAR(20) PRIMARY KEY, time DATETIME BASETIME,
open_price DOUBLE, high_price DOUBLE, low_price DOUBLE, close_price DOUBLE,
volume DOUBLE, cnt LONG, firsttime DATETIME, lasttime DATETIME
);
CREATE TAG TABLE ch6_candle_10m (
code VARCHAR(20) PRIMARY KEY, time DATETIME BASETIME,
open_price DOUBLE, high_price DOUBLE, low_price DOUBLE, close_price DOUBLE,
volume DOUBLE, cnt LONG, firsttime DATETIME, lasttime DATETIME
);
CREATE ROLLUP ch6_candle_ru_min INTO (ch6_candle_min)
AS (
SELECT code, DATE_TRUNC('minute', time) AS time,
FIRST(time, price), MAX(price), MIN(price), LAST(time, price),
SUM(volume), COUNT(*), MIN(time), MAX(time)
FROM ch6_ticks
GROUP BY code, time
) INTERVAL 1 MIN;
CREATE ROLLUP ch6_candle_ru_10m INTO (ch6_candle_10m)
AS (
SELECT code, DATE_BIN('min', 10, time, TO_DATE('2000-01-01 00:00:00')) AS time,
FIRST(firsttime, open_price), MAX(high_price),
MIN(low_price), LAST(lasttime, close_price),
SUM(volume), SUM(cnt), MIN(firsttime), MAX(lasttime)
FROM ch6_candle_min
GROUP BY code, time
) INTERVAL 10 MIN;
INSERT INTO ch6_ticks VALUES ('AAPL', TO_DATE('2026-01-01 09:00:00'), 100, 2);
INSERT INTO ch6_ticks VALUES ('AAPL', TO_DATE('2026-01-01 09:00:30'), 105, 3);
INSERT INTO ch6_ticks VALUES ('AAPL', TO_DATE('2026-01-01 09:01:00'), 103, 1);
INSERT INTO ch6_ticks VALUES ('AAPL', TO_DATE('2026-01-01 09:01:30'), 99, 4);
EXEC TABLE_FLUSH(ch6_ticks);
ALTER ROLLUP ch6_candle_ru_min FORCE;
EXEC TABLE_FLUSH(ch6_candle_min);
ALTER ROLLUP ch6_candle_ru_10m FORCE;
SELECT code, time,
FIRST(firsttime, open_price), MAX(high_price),
MIN(low_price), LAST(lasttime, close_price), SUM(volume), SUM(cnt)
FROM ch6_candle_10m
GROUP BY code, time ORDER BY code, time;
```
The 09:00 10-minute bucket has Open=100, High=105, Low=99, Close=99, volume=10, and cnt=4. NULL
prices or volumes require separate rules and tests for choosing row timestamps and values. If
multiple trades share a timestamp but have business ordering, design an additional identifier.
This 10-minute Custom example demonstrates creation and querying. Do not assume its interval is
supported by current Custom time-range REBUILD processing. Check
[REBUILD Limitations](../rollup-rebuild/).
## Status and Cleanup
Jobs start automatically after creation, so do not immediately repeat START. Call STOP/START and
FORCE according to job state. Dropping a target TAG is blocked while its job exists.
```sql
SELECT DISTINCT ROLLUP_NAME, ROLLUP_TABLE, ROOT_TABLE, EXT_TYPE,
INTERVAL_TIME, WAKEUP_INTERVAL
FROM V$ROLLUP WHERE ROLLUP_NAME = 'CH6_CUSTOM_RU';
DROP ROLLUP ch6_candle_ru_10m;
DROP ROLLUP ch6_candle_ru_min;
DROP TABLE ch6_candle_10m;
DROP TABLE ch6_candle_min;
DROP TABLE ch6_ticks;
DROP ROLLUP ch6_custom_ru;
DROP TABLE ch6_custom_dst;
DROP TABLE ch6_custom_src;
```
EXT_TYPE=2 indicates Custom; PREDICATE records the SELECT body. Clean up only objects from the
exercises you ran. For source corrections and retry/completion checks for upper-level reaggregation,
follow [Control and Status](../ingestion-control-rollup/) and the REBUILD procedure.
---
title: "6.7 Extension ROLLUP and FIRST/LAST"
url: https://docs.machbase.com/dbms/tag-rollup-usage/extension-rollup/
language: en
kind: page
---
# 6.7 Extension ROLLUP and FIRST/LAST
## EXTENSION versus Source FIRST/LAST
EXTENSION adds first/last values and associated timestamp information to ROLLUP. Using FIRST/LAST in
an ordinary source GROUP BY differs from querying FIRST/LAST from stored ROLLUP statistics. The
latter requires an applicable extension ROLLUP.
EXTENSION adds only first/last values and timestamps. MIN, MAX, SUM, COUNT, and SUMSQ, the sum of
squares, are also stored in general ROLLUP. For deriving variance and standard deviation from SUMSQ,
see [SUMSQ, Variance, and Standard Deviation](../query-syntax-rollup/#query-sumsq-stddev-rollup).
## Preparation and Ingestion
```sql
CREATE TAG TABLE ch6_ext (
code VARCHAR(20) PRIMARY KEY,
time DATETIME BASETIME,
price DOUBLE
);
CREATE ROLLUP ch6_ext_first
ON ch6_ext(price) INTERVAL 1 MIN EXTENSION;
CREATE ROLLUP ch6_ext_plain
ON ch6_ext(price) INTERVAL 1 MIN;
INSERT INTO ch6_ext VALUES ('AAPL', TO_DATE('2026-01-01 09:00:00', 'YYYY-MM-DD HH24:MI:SS'), 100);
INSERT INTO ch6_ext VALUES ('AAPL', TO_DATE('2026-01-01 09:00:10', 'YYYY-MM-DD HH24:MI:SS'), 105);
INSERT INTO ch6_ext VALUES ('AAPL', TO_DATE('2026-01-01 09:00:20', 'YYYY-MM-DD HH24:MI:SS'), 99);
INSERT INTO ch6_ext VALUES ('AAPL', TO_DATE('2026-01-01 09:00:30', 'YYYY-MM-DD HH24:MI:SS'), 103);
EXEC TABLE_FLUSH(ch6_ext);
ALTER ROLLUP ch6_ext_first FORCE;
ALTER ROLLUP ch6_ext_plain FORCE;
```
## Comparing Source and OHLC Results
```sql
SELECT DATE_TRUNC('minute', time) AS bucket,
FIRST(time, price) AS open_price, MAX(price) AS high_price,
MIN(price) AS low_price, LAST(time, price) AS close_price
FROM ch6_ext WHERE code = 'AAPL'
GROUP BY bucket ORDER BY bucket;
SELECT /*+ ROLLUP_TABLE(ch6_ext_first) */
rollup('min', 1, time) AS bucket,
FIRST(time, price) AS open_price, MAX(price) AS high_price,
MIN(price) AS low_price, LAST(time, price) AS close_price
FROM ch6_ext WHERE code = 'AAPL'
GROUP BY bucket ORDER BY bucket;
```
Both queries return Open=100, High=105, Low=99, Close=103 in the 09:00 bucket. ROLLUP FIRST/LAST use
BASETIME as the first argument and the aggregate column as the second. Design separately for
equal-timestamp values when additional business ordering is required.
## General and Extension Candidates Together
General ROLLUP does not always take priority over extension ROLLUP. Registration order affects
candidates with the same conditions and interval. This example creates extension first, but
explicitly selects it when results must depend on a particular candidate. Where only extension is
applicable, it can be selected without a hint.
The following query intentionally fails because it forces general ROLLUP.
```sql
SELECT /*+ ROLLUP_TABLE(ch6_ext_plain) */
rollup('min', 1, time) AS bucket, FIRST(time, price)
FROM ch6_ext WHERE code = 'AAPL'
GROUP BY bucket;
```
For automatic extension hierarchies, use `WITH ROLLUP (SEC) EXTENSION` in a separate table's CREATE
statement. Keep the extension attribute consistent when creating a hierarchy with FROM. For Custom
OHLCV reaggregation, see [Custom Examples](../custom-rollup/).
## Clean Up
```sql
DROP ROLLUP ch6_ext_plain;
DROP ROLLUP ch6_ext_first;
DROP TABLE ch6_ext;
```
---
title: "6.8 JSON SUMMARIZED ROLLUP"
url: https://docs.machbase.com/dbms/tag-rollup-usage/json-summarized-rollup/
language: en
kind: page
---
# 6.8 JSON SUMMARIZED ROLLUP
## JSON Path and Full-Document Aggregation
JSON path ROLLUP aggregates numeric values at a specified path. Full-document ROLLUP maintains
statistics for numeric paths in a JSON SUMMARIZED column. Do not treat missing paths, JSON null, SQL
NULL, and arrays as equivalent samples.
## Preparation
```sql
CREATE TAG TABLE ch6_json (
name VARCHAR(32) PRIMARY KEY,
time DATETIME BASETIME,
value JSON SUMMARIZED
);
CREATE ROLLUP ch6_json_metric ON ch6_json(value.metric) INTERVAL 1 MIN;
CREATE ROLLUP ch6_json_whole ON ch6_json(value) INTERVAL 1 MIN;
INSERT INTO ch6_json VALUES ('S1', TO_DATE('2026-01-01 00:00:00'),
'{"metric":10,"nested":{"x":2},"status":"OK","items":[1,2]}');
INSERT INTO ch6_json VALUES ('S1', TO_DATE('2026-01-01 00:00:10'),
'{"metric":20,"nested":{"x":4},"status":"WARN","items":[3,4]}');
INSERT INTO ch6_json VALUES ('S1', TO_DATE('2026-01-01 00:00:20'),
'{"metric":null,"status":false}');
INSERT INTO ch6_json VALUES ('S1', TO_DATE('2026-01-01 00:00:30'), NULL);
EXEC TABLE_FLUSH(ch6_json);
ALTER ROLLUP ch6_json_metric FORCE;
ALTER ROLLUP ch6_json_whole FORCE;
```
## Comparing Path Aggregates
```sql
SELECT DATE_TRUNC('minute', time) AS bucket,
COUNT(JSON_EXTRACT_DOUBLE(value, '$.metric')),
AVG(JSON_EXTRACT_DOUBLE(value, '$.metric'))
FROM ch6_json WHERE name = 'S1'
GROUP BY bucket ORDER BY bucket;
SELECT /*+ ROLLUP_TABLE(ch6_json_metric) */
rollup('min', 1, time) AS bucket,
COUNT(value.metric), AVG(value.metric)
FROM ch6_json WHERE name = 'S1'
GROUP BY bucket ORDER BY bucket;
SELECT /*+ ROLLUP_TABLE(ch6_json_metric) */
rollup('min', 1, time) AS bucket, AVG(value->'$.metric')
FROM ch6_json WHERE name = 'S1'
GROUP BY bucket ORDER BY bucket;
```
metric has two valid numeric samples with average 15. Dot and arrow syntax express the same path.
Specifying an array element in a path is different from full-document aggregation automatically
expanding arrays. `value.items[0]."metric-id"` is an example path declaration, but first verify that
the JSON structure and numeric samples actually exist.
## Full-Document Aggregation and COUNT
```sql
SELECT COUNT(*) AS raw_rows, COUNT(value) AS raw_documents FROM ch6_json;
SELECT /*+ ROLLUP_TABLE(ch6_json_whole) */
rollup('min', 1, time) AS bucket,
COUNT(value), AVG(value), MIN(value), MAX(value), SUM(value)
FROM ch6_json WHERE name = 'S1'
GROUP BY bucket ORDER BY bucket;
```
Source COUNT(*) is 4 and COUNT(value) is 3. Full-document ROLLUP COUNT(value) sums stored document
aggregate counts. Currently those counts are built from source COUNT(*), so this bucket containing
both numeric-path documents and SQL NULL returns 4. Do not apply the ordinary source COUNT(value)
NULL-exclusion rule unchanged.
AVG returns metric=15 and nested.x=3, calculated from the valid numeric samples at each path.
Strings, booleans, JSON null, and arrays are excluded from numeric aggregation; nonnumeric paths can
appear as null or be omitted. Do not compare serialized JSON as fixed strings based on key order.
Test separate samples for documents with no numeric paths and intervals containing only SQL NULL.
JSON path and full-document aggregation use different candidate modes. Do not assume forcing a
ROLLUP from another mode with a hint gives equivalent results. Invalid JSON causes an ingestion
error.
## Clean Up
```sql
DROP ROLLUP ch6_json_whole;
DROP ROLLUP ch6_json_metric;
DROP TABLE ch6_json;
```
---
title: "6.9 ROLLUP Control and State"
url: https://docs.machbase.com/dbms/tag-rollup-usage/ingestion-control-rollup/
language: en
kind: page
---
# 6.9 ROLLUP Control and State
## Job State and Processing Completion
ROLLUP starts automatically at creation. Repeating START immediately or stopping an already stopped
job can cause a state error. This exercise separates states in the sequence create → STOP → insert →
START → WAKEUP → FORCE.
```sql
CREATE TAG TABLE ch6_control (
name VARCHAR(32) PRIMARY KEY,
time DATETIME BASETIME,
value DOUBLE,
quality INTEGER
);
CREATE ROLLUP ch6_control_ru ON ch6_control(value)
INTERVAL 1 MIN WAKEUP INTERVAL 10 SEC;
ALTER ROLLUP ch6_control_ru STOP;
INSERT INTO ch6_control VALUES ('TEMP_01', TO_DATE('2026-01-01 00:00:00', 'YYYY-MM-DD HH24:MI:SS'), 10.0, 1);
INSERT INTO ch6_control VALUES ('TEMP_01', TO_DATE('2026-01-01 00:00:30', 'YYYY-MM-DD HH24:MI:SS'), 20.0, 1);
INSERT INTO ch6_control VALUES ('TEMP_01', TO_DATE('2026-01-01 00:01:00', 'YYYY-MM-DD HH24:MI:SS'), 30.0, 1);
INSERT INTO ch6_control VALUES ('TEMP_02', TO_DATE('2026-01-01 00:00:00', 'YYYY-MM-DD HH24:MI:SS'), 100.0, 1);
EXEC TABLE_FLUSH(ch6_control);
SELECT DISTINCT ROLLUP_NAME, ENABLED, INTERVAL_TIME, WAKEUP_INTERVAL
FROM V$ROLLUP WHERE ROLLUP_NAME = 'CH6_CONTROL_RU';
ALTER ROLLUP ch6_control_ru START;
ALTER ROLLUP ch6_control_ru WAKEUP;
ALTER ROLLUP ch6_control_ru FORCE;
SELECT rollup('min', 1, time) AS bucket, AVG(value)
FROM ch6_control WHERE name = 'TEMP_01'
GROUP BY bucket ORDER BY bucket;
```
While stopped, ENABLED is 0, INTERVAL_TIME is 60000 ms, and WAKEUP_INTERVAL is 10000 ms. The final
query returns TEMP_01 averages of 15 at 00:00 and 30 at 00:01.
| Command | Purpose | Completion meaning |
|---|---|---|
| STOP | Stop the job | Unprocessed input remains |
| START | Resume a stopped job | Continue from the processing position |
| WAKEUP | Wake the job | Does not wait for processing completion |
| FORCE | Wait for the target source processing range to catch up | Does not recalculate historical corrections or complete future input |
| ROLLUP_REBUILD | Recalculate historical buckets for supported targets | Rebuild aggregates after source corrections |
Instead of SQL ALTER, you can use named `EXEC ROLLUP_START(name)`, `ROLLUP_STOP(name)`, and
`ROLLUP_FORCE(name)` calls. Do not execute the same transition consecutively using both forms.
Distinguish unnamed bulk control from control of a specific job.
## WAKEUP INTERVAL
If omitted, WAKEUP INTERVAL equals the creation INTERVAL. It must be positive, no greater than the
aggregation interval, and divide that interval exactly. More frequent wakeups can reduce lag but
increase processing load.
```sql
ALTER ROLLUP ch6_control_ru SET WAKEUP INTERVAL 5 SEC;
SELECT DISTINCT ROLLUP_NAME, INTERVAL_TIME, WAKEUP_INTERVAL
FROM V$ROLLUP WHERE ROLLUP_NAME = 'CH6_CONTROL_RU';
```
WAKEUP_INTERVAL becomes 5000 ms. The next statement intentionally fails because its interval does
not divide 60 seconds.
```sql
ALTER ROLLUP ch6_control_ru SET WAKEUP INTERVAL 7 SEC;
```
## Reading V$ROLLUP
| Column | Meaning |
|---|---|
| ROLLUP_NAME | Job name used for control |
| ROLLUP_TABLE | Aggregate target table; user target TAG for Custom |
| SOURCE_TABLE, ROOT_TABLE | Direct source and root-source relationship |
| COLUMN_NAME | Column aggregated in general/path mode |
| INTERVAL_TIME, WAKEUP_INTERVAL | Creation and wakeup intervals in milliseconds |
| LAST_WAKEUP_TIME, NEXT_WAKEUP_TIME | Previous wakeup and next scheduled wakeup |
| EXT_TYPE | 0 general, 1 extension, 2 Custom |
| PREDICATE | General predicate or Custom SELECT body |
| ENABLED | Whether the job is enabled |
| RUN_STATE | `I` initial, `S` waiting, `R` processing |
| END_RID | Source processing position |
| LAST_ELAPSED_MSEC | Previous processing time in ms |
| DATABASE_NAME, USER_ID | Database and owner identifiers |
```sql
SELECT ROLLUP_NAME, ROLLUP_TABLE, SOURCE_TABLE, ROOT_TABLE,
INTERVAL_TIME, WAKEUP_INTERVAL, LAST_WAKEUP_TIME, NEXT_WAKEUP_TIME,
ENABLED, RUN_STATE, LAST_ELAPSED_MSEC
FROM V$ROLLUP WHERE ROLLUP_NAME = 'CH6_CONTROL_RU'
ORDER BY ROLLUP_NAME;
SHOW ROLLUPGAP;
```
SHOW ROLLUPGAP is a machsql-only client command; do not send it to an SDK's ordinary SQL API. GAP is
the difference between source and ROLLUP processing RIDs, not elapsed time lag. Check it alongside
all hierarchy levels and relevant nodes. gap=0 does not mean corrections to already aggregated
source data are reflected. During continuous input it changes with observation time, so pause input
for reproducible comparisons.
If retention deletes source data while a job is stopped, START cannot restore it. Run FORCE from
lower to upper levels. On failure, check the first error, state, and source accessibility.
## Clean Up
```sql
DROP ROLLUP ch6_control_ru;
DROP TABLE ch6_control;
```
For detailed command contracts, see the
[EXEC Reference](../../reference/sql/syntax/execute-procedure-syntax/) and
[ROLLUP Troubleshooting](../../troubleshooting/rollup/).
---
title: "6.10 ROLLUP_REBUILD"
url: https://docs.machbase.com/dbms/tag-rollup-usage/rollup-rebuild/
language: en
kind: page
---
# 6.10 ROLLUP_REBUILD
ROLLUP_REBUILD recalculates aggregates after historical source-value corrections and is available
only in Standard Edition. Unlike FORCE, it deletes and recreates affected buckets. It is not a
general-purpose command for rebuilding arbitrary ROLLUP definitions.
## Check Supported Targets First
| Target | Current support path |
|---|---|
| Complete SEC→MIN→HOUR hierarchy created with WITH ROLLUP | Basic numeric, extension, and full-document JSON modes |
| Supported Custom tree linked to the source | Check 1 SEC/1 MIN/1 HOUR intervals and SELECT compatibility with rebuild boundaries |
| Manually named general ROLLUP with arbitrary intervals | Do not assume the same support as automatic hierarchies |
| Automatic MIN/HOUR-only hierarchy without SEC | Not considered a complete default hierarchy |
| Custom with other intervals, such as 10 MIN | Unsupported by the current time-boundary generation path |
| Cluster Edition | Unsupported |
Custom SELECT buckets, INTERVAL, and reference time zone must align with rebuild boundaries. First
inspect the tree for unsupported jobs. Creation support for a Custom expression or interval does not
imply rebuild support.
## Time Arguments and Bucket Boundaries
Specify timestamp strings or TO_DATE with constant strings. This path does not evaluate arbitrary
DATETIME expressions; do not use NOW arithmetic or bound parameters in examples.
The operation includes the buckets containing both endpoints and expands to entire buckets. At the
1-minute level, 00:00:30–00:01:00 recalculates the 00:00 and 00:01 buckets, namely
[00:00:00, 00:02:00). Equal start/end values recalculate the containing bucket; start later than end is an error. Upper time levels expand to wider buckets.
## Default Hierarchy and Custom Correction Exercise
### 1. Preparation and Initial Aggregation
```sql
CREATE TAG TABLE ch6_rebuild (
name VARCHAR(32) PRIMARY KEY, time DATETIME BASETIME, value DOUBLE SUMMARIZED
) WITH ROLLUP (SEC);
CREATE TAG TABLE ch6_rebuild_dst (
name VARCHAR(32) PRIMARY KEY, time DATETIME BASETIME, sum_value DOUBLE, cnt LONG
);
CREATE ROLLUP ch6_rebuild_custom INTO (ch6_rebuild_dst)
AS (
SELECT name, DATE_TRUNC('minute', time) AS time, SUM(value), COUNT(value)
FROM ch6_rebuild GROUP BY name, time
) INTERVAL 1 MIN;
INSERT INTO ch6_rebuild VALUES ('S1', TO_DATE('2026-01-01 00:00:00'), 1);
INSERT INTO ch6_rebuild VALUES ('S1', TO_DATE('2026-01-01 00:00:30'), 200);
INSERT INTO ch6_rebuild VALUES ('S1', TO_DATE('2026-01-01 00:01:00'), 300);
INSERT INTO ch6_rebuild VALUES ('S1', TO_DATE('2026-01-01 00:01:30'), 4);
EXEC TABLE_FLUSH(ch6_rebuild);
SELECT DISTINCT ROLLUP_NAME, INTERVAL_TIME FROM V$ROLLUP
WHERE ROOT_TABLE = 'CH6_REBUILD' ORDER BY INTERVAL_TIME, ROLLUP_NAME;
ALTER ROLLUP _CH6_REBUILD_ROLLUP_SEC FORCE;
ALTER ROLLUP _CH6_REBUILD_ROLLUP_MIN FORCE;
ALTER ROLLUP _CH6_REBUILD_ROLLUP_HOUR FORCE;
ALTER ROLLUP ch6_rebuild_custom FORCE;
SELECT rollup('min', 1, time) AS bucket, AVG(value)
FROM ch6_rebuild WHERE name = 'S1'
GROUP BY bucket ORDER BY bucket;
```
Verify that automatically generated names in control commands match V$ROLLUP above. Do not guess
names of other objects. Initial minute averages are 100.5 and 152.
### 2. Correct Source Data and Rebuild
```sql
UPDATE ch6_rebuild SET value = 20
WHERE name = 'S1' AND time = TO_DATE('2026-01-01 00:00:30');
UPDATE ch6_rebuild SET value = 30
WHERE name = 'S1' AND time = TO_DATE('2026-01-01 00:01:00');
EXEC ROLLUP_REBUILD(ch6_rebuild, 'S1',
TO_DATE('2026-01-01 00:00:30'),
TO_DATE('2026-01-01 00:01:00'));
```
### 3. Compare Source, General, and Custom Results
```sql
SELECT DATE_TRUNC('minute', time) AS bucket, SUM(value), COUNT(value), AVG(value)
FROM ch6_rebuild WHERE name = 'S1'
GROUP BY bucket ORDER BY bucket;
SELECT rollup('min', 1, time) AS bucket, SUM(value), COUNT(value), AVG(value)
FROM ch6_rebuild WHERE name = 'S1'
GROUP BY bucket ORDER BY bucket;
SELECT time, SUM(sum_value), SUM(cnt), SUM(sum_value) / SUM(cnt)
FROM ch6_rebuild_dst WHERE name = 'S1'
GROUP BY time ORDER BY time;
SHOW ROLLUPGAP;
```
All three results have sum 21, count 2, average 10.5 at 00:00, and sum 34, count 2, average 17 at
00:01. The value 4 at 00:01:30, after the end argument, must also be included when its bucket is
recalculated.
## Operational Impact and Failure Recovery
Related jobs catch up, stop, recalculate, and restart. Internal processing also stabilizes source
reads, so normal work does not simply continue unchanged during rebuilding. Measure acceptable
query/ingestion delays and downtime in a validation environment first.
Do not assume all stages roll back as one atomic transaction. State recovery after failure is best
effort; check actual data and V$ROLLUP. Restoring the original stopped state is not guaranteed
either. Before retrying, verify supported targets, remaining source data, and reaggregation scope.
A nonexistent tag can be a no-op when a valid rebuild target is otherwise prepared. A successful
response alone does not prove the intended tag was processed; compare actual results. Original
statistics cannot be reconstructed after the source data has been deleted.
## Clean Up
```sql
DROP ROLLUP ch6_rebuild_custom;
DROP TABLE ch6_rebuild_dst;
DROP TABLE ch6_rebuild CASCADE;
```
Remove Custom targets separately. If a general definition must change or the task is outside this
procedure's scope, prepare a supported new definition and data migration procedure. Do not
substitute arbitrary deletion of internal storage tables. For exact arguments, see the
[REBUILD Reference](../../reference/sql/syntax/rollup-rebuild-syntax/).
---
title: "6.11 ROLLUP Performance Tuning"
url: https://docs.machbase.com/dbms/tag-rollup-usage/performance-tuning-rollup/
language: en
kind: page
---
# 6.11 ROLLUP Performance Tuning
## Compare Equivalent Results Before Costs
ROLLUP reduces the number of raw rows read. First verify equivalent tags, time ranges, NULL
handling, and aggregation criteria; then compare execution time, CPU, I/O, and memory. Small-example
timings do not guarantee production performance.
## Comparison Exercise
```sql
CREATE TAG TABLE ch6_perf (
name VARCHAR(32) PRIMARY KEY,
time DATETIME BASETIME,
value DOUBLE,
quality INTEGER
);
CREATE ROLLUP ch6_perf_ru ON ch6_perf(value) INTERVAL 1 MIN;
INSERT INTO ch6_perf VALUES ('TEMP_01', TO_DATE('2026-01-01 00:00:00', 'YYYY-MM-DD HH24:MI:SS'), 10.0, 1);
INSERT INTO ch6_perf VALUES ('TEMP_01', TO_DATE('2026-01-01 00:00:30', 'YYYY-MM-DD HH24:MI:SS'), 20.0, 1);
INSERT INTO ch6_perf VALUES ('TEMP_01', TO_DATE('2026-01-01 00:01:00', 'YYYY-MM-DD HH24:MI:SS'), 30.0, 1);
INSERT INTO ch6_perf VALUES ('TEMP_02', TO_DATE('2026-01-01 00:00:00', 'YYYY-MM-DD HH24:MI:SS'), 100.0, 1);
EXEC TABLE_FLUSH(ch6_perf);
ALTER ROLLUP ch6_perf_ru FORCE;
SELECT DATE_TRUNC('minute', time) AS bucket,
SUM(value), COUNT(value), AVG(value), MIN(value), MAX(value)
FROM ch6_perf
WHERE name = 'TEMP_01'
AND time >= TO_DATE('2026-01-01 00:00:00')
AND time < TO_DATE('2026-01-01 00:02:00')
GROUP BY bucket ORDER BY bucket;
SELECT rollup('min', 1, time) AS bucket,
SUM(value), COUNT(value), AVG(value), MIN(value), MAX(value)
FROM ch6_perf
WHERE name = 'TEMP_01'
AND time >= TO_DATE('2026-01-01 00:00:00')
AND time < TO_DATE('2026-01-01 00:02:00')
GROUP BY bucket ORDER BY bucket;
EXPLAIN SELECT rollup('min', 1, time) AS bucket, AVG(value)
FROM ch6_perf WHERE name = 'TEMP_01'
GROUP BY bucket;
```
Both results show sum 30, count 2, average 15, minimum 10, maximum 20 at 00:00; and sum 30, count 1,
average 30, minimum 30, maximum 30 at 00:01. Inspect the source DATE_TRUNC execution plan the same
way.
## Measuring Production Load
| Metric | Conditions to record |
|---|---|
| Ingestion throughput | Tag count, input rate, row width, and concurrent writers |
| Query latency | Tag range, buckets, concurrent queries, and cold/warm cache state |
| Aggregation lag | Gap, job state, and processing time at each level |
| Storage | Source data, aggregates, indexes, compression, and replicas |
| Change impact | Ingestion/query costs before and after wakeup or hierarchy changes |
Shorter WAKEUP intervals can create partial aggregates more frequently within the same bucket.
Smaller aggregation buckets increase storage and reaggregation volume. Treat the two intervals as
distinct tuning parameters. Measure concurrent ingestion and queries as well as isolated runs.
## Meaning of Hierarchy Size
The following logical bucket counts assume one tag has data in every interval for 30 days.
| Query interval | Bucket count |
|---|---:|
| 1 second | 2,592,000 |
| 1 minute | 43,200 |
| 1 hour | 720 |
| 1 day | 30 |
These are not physical row counts. Measure actual storage with sample loading, including partial
aggregates, empty intervals, NULLs, and conditional filters. The coarsest candidate does not always
produce correct results; also check required resolution, origin, and candidate constraints.
Daily results reaggregate applicable HOUR/MIN/SEC statistics. A 24 HOUR ROLLUP is not recommended as
a substitute storage level for `rollup('day', 1, ...)`. Follow
[Query Candidate Rules](../query-syntax-rollup/) and actual EXPLAIN output.
## Distinguishing Lag from Mismatches
gap=0 means processing positions have caught up, not that source corrections have been reflected.
Before comparing performance, distinguish aggregation progress from historical correction state and
check matching definitions and filters. Do not use `rollup()` without an applicable ROLLUP to
benchmark source queries.
## Clean Up
```sql
DROP ROLLUP ch6_perf_ru;
DROP TABLE ch6_perf;
```
Choose the next adjustment using [Control and Status](../ingestion-control-rollup/),
[Hierarchy Design](../target-tag-table-design/), and
[Troubleshooting](../../troubleshooting/rollup/).
---
title: "6.12 ROLLUP Scenarios"
url: https://docs.machbase.com/dbms/tag-rollup-usage/patterns-scenarios/
language: en
kind: page
---
# 6.12 ROLLUP Scenarios
## Source Data and Interval Statistics by Sensor
Do not combine averages for sensors with different units, even if they measure at the same time.
This standalone exercise manages current units as metadata and queries aggregates by tag.
### 1. Create and Insert
```sql
CREATE TAG TABLE ch6_scenario (
name VARCHAR(32) PRIMARY KEY, time DATETIME BASETIME, value DOUBLE
) METADATA (unit VARCHAR(16));
INSERT INTO ch6_scenario METADATA VALUES ('TEMP_01', 'celsius');
INSERT INTO ch6_scenario METADATA VALUES ('PRESS_01', 'bar');
INSERT INTO ch6_scenario VALUES ('TEMP_01', TO_DATE('2026-01-01 10:00:00'), 20);
INSERT INTO ch6_scenario VALUES ('TEMP_01', TO_DATE('2026-01-01 10:00:30'), 22);
INSERT INTO ch6_scenario VALUES ('PRESS_01', TO_DATE('2026-01-01 10:00:00'), 1.02);
CREATE ROLLUP ch6_scenario_ru ON ch6_scenario(value) INTERVAL 1 MIN;
EXEC TABLE_FLUSH(ch6_scenario);
ALTER ROLLUP ch6_scenario_ru FORCE;
```
A ROLLUP created after source ingestion still initially aggregates remaining data. Creation
completion alone does not mean initial aggregation is finished.
### 2. Check Results
```sql
SELECT name, unit FROM ch6_scenario METADATA ORDER BY name;
SELECT name, DATE_TRUNC('minute', time) AS bucket, COUNT(value), AVG(value)
FROM ch6_scenario GROUP BY name, bucket ORDER BY name, bucket;
SELECT name, rollup('min', 1, time) AS bucket, COUNT(value), AVG(value)
FROM ch6_scenario GROUP BY name, bucket ORDER BY name, bucket;
SHOW ROLLUPGAP;
```
TEMP_01 has 2 samples averaging 21°C; PRESS_01 has 1 sample averaging 1.02 bar. Metadata queries
return current units and do not automatically preserve historical unit changes.
### 3. Inspect Individual Observations
The data uses fixed timestamps, so query the same fixed range. Applying a current-time “last 5
minutes” predicate can return no rows depending on the execution date.
```sql
SELECT name, time, value FROM ch6_scenario
WHERE name = 'TEMP_01'
AND time >= TO_DATE('2026-01-01 10:00:00')
AND time < TO_DATE('2026-01-01 10:01:00')
ORDER BY time;
```
The two source values are 20 and 22. Their individual values or occurrence order cannot be
reconstructed from the aggregate average 21.
### 4. Clean Up
```sql
DROP ROLLUP ch6_scenario_ru;
DROP TABLE ch6_scenario;
```
## Applying ROLLUP to Workloads
| Requirement | Design to check |
|---|---|
| Valid-quality statistics | Fix filters and candidate hints; compare with source data |
| OHLC | Extension FIRST/LAST or Custom reaggregation with auxiliary timestamps |
| Multiple-sensor comparison | Tag-specific units and matching buckets/query ranges |
| Consumption from cumulative meters | Boundary differences and reset/replacement/missing-data rules; distinguish from sample averages |
| Availability | Distinguish sample ratios from time ratios; define missing-interval policy |
| Combining recent source data with long-term aggregates | Separate nonoverlapping ranges at a verified aggregation-completion point |
When combining source and ROLLUP results with UNION ALL, check boundary duplicates, omissions, and
differing sample counts. Do not assume a fixed lag such as the latest two minutes always being
unaggregated. Pass sums and valid counts when recombining partial results into averages.
Complete feature exercises are available for [Conditional](../conditional-rollup/),
[Extension](../extension-rollup/), [JSON](../json-summarized-rollup/), and
[Custom](../custom-rollup/) ROLLUP. If problems arise, first check state and semantics using the
[Diagnostic Sequence](../../troubleshooting/rollup/).
---
title: "7. LOG Table Usage"
url: https://docs.machbase.com/dbms/log-table-usage/
language: en
kind: section
---
# 7. LOG Table Usage
Storing logs and finding the logs you need are different tasks. During incident analysis, questions
often arise that were overlooked during ingestion: does this timestamp represent occurrence or
collection, and why does a word in a message not appear in search results?
This chapter connects LOG table selection and design with ingestion, search, and retention
management. It explains both how to run commands and how to check their results. LOG uses a model
that continuously appends source events, so its usage differs from business tables whose stored rows
are repeatedly updated.
## Chapter Contents
If you are new to LOG tables, read the overview and schema sections, then follow creation,
ingestion, and querying in order. Start with 7.10 for time predicates or 7.11 for message search.
| Section | What you will learn |
|---|---|
| [7.1 Overview and Use Criteria](./overview-use-criteria/) | Distinguish LOG use cases from other table types |
| [7.2 Table Structure and Schema](./table-structure-schema/) | Separate event time, search fields, and raw messages |
| [7.3 Create, Alter, and Drop](./create-alter-drop/) | Change schemas while checking existing data |
| [7.4 Data Ingestion](./data-input-mutation/) | Choose INSERT, Append, or file loading |
| [7.5 Queries and Analysis](./query-analysis/) | Query time ranges and join reference data |
| [7.6 Indexes and Performance](./index-performance/) | Choose indexes and inspect execution plans |
| [7.7 Operations and Data Lifecycle](./operations-lifecycle/) | Check deletion boundaries and apply retention policies |
| [7.8 Constraints, Errors, and Troubleshooting](./constraints-errors-troubleshooting/) | Identify causes and remedies from symptoms |
| [7.9 Usage Patterns and Scenarios](./patterns-scenarios/) | Ingest, search, and aggregate application logs |
| [7.10 _arrival_time Time Model](./arrival-time-model/) | Distinguish automatic, explicit, and out-of-order timestamps |
| [7.11 Text Search and KEYWORD Indexes](./text-search-keyword-index/) | Understand how search methods affect results |
| [7.12 Network Type Queries](./regex-network-query/) | Query IPV4/IPV6 addresses and ranges |
## Example Environment
The examples follow the DBMS 8.7 documentation. Unless stated otherwise, SQL exercises target a
Standard Edition validation environment. Use an account that can create tables and indexes. For
Cluster environments, also review
[Edition Differences](/dbms/reference/support-scope-constraints/edition/) and the relevant
operational procedures.
Example objects start with `ch7_`. Each section creates its own tables and can be followed
independently. Before rerunning a section, confirm that its final cleanup SQL was executed.
Intentionally failing SQL is separated from the normal exercise.
Caution: `DELETE`, `TRUNCATE`, and `DROP` remove data. Do not substitute production table names for
the example names.
If your results differ, compare the SQL you ran with its actual results. Identifying the first step
that differs helps narrow down the cause.
---
title: "7.1 Overview and Use Criteria"
url: https://docs.machbase.com/dbms/log-table-usage/overview-use-criteria/
language: en
kind: page
---
# 7.1 Overview and Use Criteria
Data with timestamps does not all need the same table type. Periodic temperature readings and a
device reconnection event require different analysis. When choosing LOG, start with what each row
represents rather than whether it has a time column.
## LOG Table Characteristics
LOG suits data that appends individual events, such as application errors, security-device block
records, and job start/end history. An `_arrival_time` column is created automatically in addition
to user columns, and queries can combine time-range predicates with message search.
The following example shows the difference between event time and ingestion time.
```sql
CREATE LOG TABLE ch7_overview (
event_time DATETIME,
device VARCHAR(32),
message VARCHAR(128)
);
INSERT INTO ch7_overview VALUES (
TO_DATE('2026-01-01 10:00:00', 'YYYY-MM-DD HH24:MI:SS'),
'DEV-01', 'connection restored'
);
SELECT _arrival_time, event_time, device, message FROM ch7_overview;
DROP TABLE ch7_overview;
```
The query returns one row. `event_time` is the fixed timestamp supplied in the example;
`_arrival_time` is assigned automatically during this insert. Running the example on another date
does not change `event_time`. For explicit timestamps and adjustment rules, see
[Time Model](../arrival-time-model/).
## Use Criteria
Consider LOG when you frequently search for errors in the last hour, requests from an IP address, or
messages containing timeout, and do not need to update stored rows. Use the Append API for
continuous ingestion and loading tools for initial data or batches of files.
The inability to update rows does not by itself make audit data tamper-proof. Design deletion
privileges, access controls, retention policies, and backups separately.
Pay particular attention to retransmission. Resending an event may store another row. LOG has no
PRIMARY KEY or UNIQUE constraints, so do not expect automatic deduplication. Retain the source event
ID and define retry and duplicate-handling policies during collection.
## Comparison with Other Tables
| Main workload | Table to consider first | Decision criteria |
|---|---|---|
| Measurements by sensor name and time-series aggregates | TAG | Name/time-axis queries and ROLLUP |
| Current reference data, such as device names and installation locations | LOOKUP | Query and modify small reference datasets |
| Order status changes, row deletion, and transaction processing | TRANSACTION | Modify rows and process operations as transactions |
| In-memory state that may be lost on restart | VOLATILE | Keep separate from source logs requiring permanent retention |
LOG does not support general `UPDATE` or `DELETE WHERE` with arbitrary predicates. You can append
correction events, but must store the original ID and correction reason and decide in the
application which corrections queries apply. If updates are routine, another table type is a clearer
choice.
## Design Criteria
First distinguish whether analysis requires event time or collection time. Then identify frequently
used predicates, such as device, severity, and IP address. Store these values in separate columns
instead of extracting them from raw messages for every query; this makes SQL easier to understand
and maintain. Define retention at this stage as well. Disk usage continues to grow if you prepare
ingestion but postpone deletion.
The next section, [Schema Design](../table-structure-schema/), translates these requirements into
columns. If the choice is unclear, compare representative events alongside the queries you expect to
run.
---
title: "7.2 Table Structure and Schema"
url: https://docs.machbase.com/dbms/log-table-usage/table-structure-schema/
language: en
kind: page
---
# 7.2 Table Structure and Schema
Putting an entire message in one column makes it quick to start collection. However, repeatedly
parsing raw messages to count errors by device makes queries complex. Retain the raw message and
extract values used repeatedly for queries and aggregates into separate columns.
## Column Layout
The following standalone example stores and checks one security event.
```sql
CREATE LOG TABLE ch7_schema (
event_time DATETIME,
event_id VARCHAR(64),
device VARCHAR(32),
severity SHORT,
src_ip IPV4,
dst_port INTEGER,
message TEXT
);
INSERT INTO ch7_schema VALUES (
TO_DATE('2026-01-01 10:00:00', 'YYYY-MM-DD HH24:MI:SS'),
'evt-0001', 'FW-01', 3, '192.0.2.10', 65535,
'connection blocked by policy'
);
SELECT event_id, device, severity, src_ip, dst_port, message
FROM ch7_schema;
```
The query returns the row `evt-0001` with port `65535`. Here, `event_id` is a tracking value, not a
key constraint that prevents duplicate ingestion.
Store the event timestamp recorded by the source in `event_time`. Do not redeclare the automatic
`_arrival_time` column in DDL. Network delays and batch migration can naturally cause the two
timestamps to differ.
## Choosing Data Types
| Value | Type | What to check |
|---|---|---|
| Device name, short code, or event ID | `VARCHAR(n)` | LOG permits 1–32,767 bytes, not characters |
| Long raw message | `TEXT` | Maximum 64 MiB; sorting and grouping the raw value are unsupported |
| Severity or small code | `SHORT` or `INTEGER` | Use consistent code meanings in collectors and queries |
| Port | `INTEGER` | `USHORT` reserves 65535 for NULL and cannot represent the full port range |
| Cumulative byte count | `LONG` | Check the expected maximum and reserved NULL value |
| Address | `IPV4` or `IPV6` | Distinguish address format from the need to retain the source string |
In LOG tables, both VARCHAR and TEXT support word search through KEYWORD indexes. Full-text search
alone is no reason to use TEXT for short codes.
String length is a common source of mistakes. `VARCHAR(100)` does not mean 100 Korean characters.
Check the byte length of UTF-8 samples and send oversized values through the actual ingestion path.
Verify whether excess length causes truncation or an error with the SDK or loading tool you will
use.
## Columns for Sorting and Aggregation
In this schema, `device` and `severity` are query predicates and aggregation keys; `message` is used
to read raw text or search for words. Applying `ORDER BY message` or `GROUP BY message` directly to
TEXT causes an error. Choose the device, error code, or event timestamp you actually need instead of
sorting entire messages.
A KEYWORD index is not a morphological analyzer or a relevance-ranked search engine. For the
difference between word search and raw substring search, see
[Text Search](../text-search-keyword-index/).
## Schema Changes and Ingestion Mappings
LOG supports adding, dropping, and renaming columns, and limited attribute changes. This does not
mean that stored row values can be changed with UPDATE. DDL changes particularly affect Appenders
that send values by column position and CSV mappings. Coordinate schema changes with ingestion
program deployment.
Clean up the example table, then continue to [Column Changes](../create-alter-drop/).
```sql
DROP TABLE ch7_schema;
```
If you are unsure which values to extract into columns, list the questions your queries must answer.
Expressing those questions in SQL makes the required columns clearer.
---
title: "7.3 Create, Alter, and Drop"
url: https://docs.machbase.com/dbms/log-table-usage/create-alter-drop/
language: en
kind: page
---
# 7.3 Create, Alter, and Drop
Adding a column takes little SQL, but production changes also require checking which values appear
in existing rows and whether existing ingestion programs continue to work. This section changes a
schema with data already present and checks the results.
## Creating a LOG Table
`CREATE TABLE` without a table type creates a TRANSACTION table. Use `CREATE LOG TABLE` in this exercise.
```sql
CREATE LOG TABLE ch7_ddl (
event_id INTEGER,
category VARCHAR(32),
severity SHORT,
message VARCHAR(128)
);
INSERT INTO ch7_ddl VALUES (1, 'network', 3, 'connection timeout');
```
Do not redeclare the automatic `_arrival_time` column in DDL. Use a separate DATETIME column for
actual event time. LOG does not support PRIMARY KEY or UNIQUE constraints.
## Adding Columns and Defaults
```sql
ALTER TABLE ch7_ddl ADD COLUMN (host_name VARCHAR(64));
ALTER TABLE ch7_ddl ADD COLUMN (source_kind VARCHAR(16) DEFAULT 'agent');
ALTER TABLE ch7_ddl ADD COLUMN (channels INT32[3] DEFAULT [1, NULL, 3]);
SELECT event_id, host_name, source_kind, channels
FROM ch7_ddl
ORDER BY event_id;
```
For existing row 1, `host_name` is NULL, `source_kind` is `agent`, and `channels` is `[1, NULL, 3]`.
This shows the difference between columns with and without DEFAULT. An ARRAY DEFAULT must have the
same number of elements as the declared length.
Next, rename a column and increase its string length.
```sql
ALTER TABLE ch7_ddl RENAME COLUMN category TO event_category;
ALTER TABLE ch7_ddl MODIFY COLUMN (message VARCHAR(4096));
ALTER TABLE ch7_ddl MODIFY COLUMN severity SET MINMAX_CACHE_SIZE = 1048576;
SELECT event_id, event_category, severity, message FROM ch7_ddl;
```
Existing values are retained. After renaming, queries must also use `event_category`. The MINMAX
example specifies 1 MiB for a numeric column; this is not a recommended value for every table.
Be careful with type changes. Extending a VARCHAR length does not convert TEXT to VARCHAR.
`MINMAX_CACHE_SIZE` also cannot be set on variable-length columns such as VARCHAR or TEXT.
## NOT NULL and Existing Data
Because `event_category` currently contains a value, the following change is allowed.
```sql
ALTER TABLE ch7_ddl MODIFY COLUMN event_category NOT NULL;
ALTER TABLE ch7_ddl MODIFY COLUMN event_category NULL;
```
`NOT NULL` without an option checks existing rows. It cannot be applied to `host_name`, which
contains NULL. Run the following SQL separately only if you want to verify the failure.
```sql
-- Expected failure: host_name is NULL in an existing row.
ALTER TABLE ch7_ddl MODIFY COLUMN host_name NOT NULL;
```
`NOT NULL NOCHECK` skips the check for existing NULL values. It neither fills existing NULL values
nor guarantees that historical data satisfies the constraint. The normal exercise does not use it.
## Dropping Indexes and Columns
```sql
CREATE INDEX ch7_ddl_host_idx ON ch7_ddl(host_name) INDEX_TYPE LSM;
DROP INDEX ch7_ddl_host_idx;
ALTER TABLE ch7_ddl DROP COLUMN (host_name);
ALTER TABLE ch7_ddl DROP COLUMN (source_kind);
ALTER TABLE ch7_ddl DROP COLUMN (channels);
SELECT event_id, event_category, severity, message FROM ch7_ddl;
```
Drop an index before dropping a column it references. Internal columns `_ARRIVAL_TIME` and `_RID`
cannot be dropped, renamed, or have their attributes changed. At least one user column must remain.
A VARCHAR length can only be increased and must not exceed 32,767 bytes.
## Deleting Data and Dropping Tables
Caution: the following commands remove example data. LOG data cannot be recovered with a TRANSACTION
table `ROLLBACK`.
```sql
TRUNCATE TABLE ch7_ddl;
SELECT COUNT(*) AS remaining_rows FROM ch7_ddl;
DROP TABLE ch7_ddl;
```
After TRUNCATE, the count is 0 and the definition remains. The final DROP also removes the
definition. To remove only older data, use [Retention Deletion](../operations-lifecycle/).
## Operational DDL Considerations
Before a change, coordinate ingestion and DDL timing. Afterward, check column names, order, and
types in SQL, Appenders, and file mappings. If a resource-in-use error occurs, check which jobs use
the table before retrying.
If the failing change is unclear, collect the original DDL and the SQL that failed. Together they
help narrow down the cause.
---
title: "7.4 Data Input"
url: https://docs.machbase.com/dbms/log-table-usage/data-input-mutation/
language: en
kind: page
---
# 7.4 Data Input
Successfully inserting one or two rows does not complete ingestion preparation. Continuous
collection must account for send buffers, partial row failures, and retransmission after
disconnection. First verify columns and timestamps with SQL, then choose an ingestion path suited to
the required throughput.
## SQL INSERT
```sql
CREATE LOG TABLE ch7_input (
event_time DATETIME,
event_id VARCHAR(32),
device VARCHAR(32),
message VARCHAR(128)
);
INSERT INTO ch7_input(event_time, event_id, device, message)
VALUES (TO_DATE('2026-01-01 10:00:00', 'YYYY-MM-DD HH24:MI:SS'),
'evt-001', 'DEV-01', 'connection timeout');
SELECT _arrival_time, event_time, event_id, device, message
FROM ch7_input;
```
The query returns one row, with `event_time` set to the fixed event timestamp. Because
`_arrival_time` is omitted, the ingestion path uses server time. Under the default setting,
out-of-order timestamps may be adjusted, so do not assume this value always exactly matches
reception time. For details, see [Time Model](../arrival-time-model/).
Next, check what happens when the same event is inserted again.
```sql
INSERT INTO ch7_input(event_time, event_id, device, message)
VALUES (TO_DATE('2026-01-01 10:00:00', 'YYYY-MM-DD HH24:MI:SS'),
'evt-001', 'DEV-01', 'connection timeout');
SELECT event_id, COUNT(*) AS received_rows
FROM ch7_input
GROUP BY event_id;
DROP TABLE ch7_input;
```
The count for `evt-001` is 2. LOG does not deduplicate events with the same name. Successful
transmission by the collector and exactly-once storage of a source event are separate concerns.
## Choosing an Ingestion Path
| Situation | Initial choice | Also check |
|---|---|---|
| Small inserts or functional checks | SQL INSERT | Column list, types, and date format |
| Continuous bulk ingestion from applications | SDK Append | Buffer transmission, row-level failures, and reconnection policy |
| CSV read by a client | csvimport or machloader | Column mapping and rejected-row files |
| Load files accessible to the server | LOAD DATA INFILE | Server path and file access permissions |
SQL INSERT incurs processing overhead for each statement. For continuous bulk ingestion, consider
the Append API, which sends rows in batches. For runnable code in your language, see
[Development and Application Integration](/dbms/development-tools-integration/).
## Append Transmission and Error Handling
After a row is passed to an Appender, it may still be in a client buffer. Check the SDK's flush and
close behavior, and handle remaining buffers and connections on exception paths as well as normal
shutdown.
A successful call does not necessarily mean every row was stored. SDKs expose results differently,
through return values, error callbacks, or success/failure counts at close. First test a small batch
deliberately containing oversized values, NULL values, and date-conversion errors.
A common problem is disconnection before a response arrives. A retry may resend a batch already
stored, so record source event IDs and processing positions. LOG INSERT and Append operations are
also outside the scope of ROLLBACK in TRANSACTION table transactions.
## File Loading and Mapping
Prepare samples containing Korean text, empty strings, NULL values, long messages, and different
time zones. Check the source field count and target column order before processing the full file.
Retain rejected-row files and logs so the same errors can be investigated later.
For complete commands, see
[Data Ingestion, Loading, and Export](/dbms/development-tools-integration/data-input-load-export/).
To preserve `_arrival_time` during historical migration, check both sort order and existing target
data. For ordinary collection, it is safer to store historical event timestamps in a separate
`event_time` column.
If ingestion fails, start with one failing source row rather than the full batch. Compare its field
values, target types, and ingestion API to identify the cause.
---
title: "7.5 Query and Analysis"
url: https://docs.machbase.com/dbms/log-table-usage/query-analysis/
language: en
kind: page
---
# 7.5 Query and Analysis
Small changes to a time range can change the row count. When aggregating consecutive days, including
the endpoint in both intervals counts boundary rows twice. This section uses fixed timestamps to
check ranges and result order, then joins reference data.
## Prepare Example Data
```sql
CREATE LOG TABLE ch7_query (
event_id INTEGER,
device VARCHAR(32),
value DOUBLE
);
INSERT INTO ch7_query(_arrival_time, event_id, device, value)
VALUES (TO_DATE('2026-01-01 10:00:00', 'YYYY-MM-DD HH24:MI:SS'), 1, 'DEV-01', 10);
INSERT INTO ch7_query(_arrival_time, event_id, device, value)
VALUES (TO_DATE('2026-01-01 11:00:00', 'YYYY-MM-DD HH24:MI:SS'), 2, 'DEV-01', 20);
INSERT INTO ch7_query(_arrival_time, event_id, device, value)
VALUES (TO_DATE('2026-01-01 11:00:00', 'YYYY-MM-DD HH24:MI:SS'), 3, 'DEV-02', 30);
INSERT INTO ch7_query(_arrival_time, event_id, device, value)
VALUES (TO_DATE('2026-01-01 12:00:00', 'YYYY-MM-DD HH24:MI:SS'), 4, 'DEV-02', 40);
SELECT _arrival_time, event_id, device, value
FROM ch7_query
ORDER BY _arrival_time, event_id;
```
Results appear in the order 1, 2, 3, 4. Because rows 2 and 3 share a timestamp, include the event
number as well as time in ORDER BY to guarantee the order. This number is managed explicitly in the
example, not an automatic LOG unique key.
## Consecutive Intervals and Time Boundaries
```sql
SELECT event_id, value
FROM ch7_query
WHERE _arrival_time >= TO_DATE('2026-01-01 10:00:00', 'YYYY-MM-DD HH24:MI:SS')
AND _arrival_time < TO_DATE('2026-01-01 12:00:00', 'YYYY-MM-DD HH24:MI:SS')
ORDER BY event_id;
```
Rows 1, 2, and 3 are selected. Starting the next interval at 12:00 counts row 4 only in that
interval. `BETWEEN` includes both endpoints and therefore has different semantics. Use the same
WHERE pattern when analyzing a user-defined `event_time` column.
## DURATION Queries
```sql
SELECT event_id FROM ch7_query
DURATION 1 HOUR BEFORE TO_DATE('2026-01-01 12:00:00', 'YYYY-MM-DD HH24:MI:SS')
ORDER BY event_id;
SELECT event_id FROM ch7_query
DURATION 1 HOUR AFTER TO_DATE('2026-01-01 10:00:00', 'YYYY-MM-DD HH24:MI:SS')
ORDER BY event_id;
SELECT event_id FROM ch7_query
DURATION FROM TO_DATE('2026-01-01 10:00:00', 'YYYY-MM-DD HH24:MI:SS')
TO TO_DATE('2026-01-01 12:00:00', 'YYYY-MM-DD HH24:MI:SS')
ORDER BY event_id;
```
| Query | Time range | Selected event_id values |
|---|---|---|
| 1 hour before 12:00 | 11:00–12:00, inclusive | 2, 3, 4 |
| 1 hour after 10:00 | 10:00–11:00, inclusive | 1, 2, 3 |
| From 10:00 to 12:00 | Both endpoints included | 1, 2, 3, 4 |
If the reference time is omitted, as in `DURATION 1 HOUR`, the current time is used. This may return
no rows for an exercise with old fixed timestamps. Place DURATION after WHERE and before GROUP BY or
ORDER BY.
Do not assume DURATION applies to every time column. It is LOG-specific and uses `_arrival_time`.
Use WHERE for TAG time columns or user-defined DATETIME predicates. For syntax details, see
[Relative Time and DURATION Dictionary](/dbms/reference/sql/relative-time/#log-duration).
## Scan Direction and Sorting
DURATION BEFORE reads from newest to oldest; AFTER reads from oldest to newest. FROM … TO changes
direction according to the order of its two timestamps. The following examples check a reversed
range and a range whose endpoints are equal.
```sql
SELECT event_id FROM ch7_query
DURATION FROM TO_DATE('2026-01-01 12:00:00', 'YYYY-MM-DD HH24:MI:SS')
TO TO_DATE('2026-01-01 10:00:00', 'YYYY-MM-DD HH24:MI:SS')
ORDER BY event_id DESC;
SELECT event_id FROM ch7_query
DURATION FROM TO_DATE('2026-01-01 11:00:00', 'YYYY-MM-DD HH24:MI:SS')
TO TO_DATE('2026-01-01 11:00:00', 'YYYY-MM-DD HH24:MI:SS')
ORDER BY event_id;
```
The first result is ordered 4, 3, 2, 1; the second, with equal endpoints, returns 2 and 3. Explicit
ORDER BY clauses fix the output order independently of scan direction.
Use ORDER BY whenever the final result order matters, including results with aggregates or joins. In
particular, do not infer which rows will be returned from LIMIT alone.
The global `TABLE_SCAN_DIRECTION` setting can affect other queries. Do not start by changing server
settings to alter display order. Check the execution plan in
[Query Tuning](/dbms/performance-tuning/performance-query-tuning/) before adjusting access paths.
## Joining LOOKUP Tables
```sql
CREATE LOOKUP TABLE ch7_query_device (
device VARCHAR(32) PRIMARY KEY,
label VARCHAR(64)
);
INSERT INTO ch7_query_device VALUES ('DEV-01', 'Boiler');
INSERT INTO ch7_query_device VALUES ('DEV-02', 'Pump');
SELECT q.event_id, d.label, q.value
FROM ch7_query q
JOIN ch7_query_device d ON q.device = d.device
WHERE q._arrival_time >= TO_DATE('2026-01-01 10:00:00', 'YYYY-MM-DD HH24:MI:SS')
AND q._arrival_time < TO_DATE('2026-01-01 12:00:00', 'YYYY-MM-DD HH24:MI:SS')
ORDER BY q.event_id;
```
The results are `(1, Boiler, 10)`, `(2, Boiler, 20)`, and `(3, Pump, 30)`. Because this query
combines LOG and LOOKUP, it uses a WHERE range on the LOG column instead of DURATION. This INNER
JOIN also excludes events without matching reference data.
Joining current LOOKUP values does not reconstruct a device description from the time of an event.
For historical descriptions, store the description in the event or design a separate history with
validity periods.
```sql
DROP TABLE ch7_query_device;
DROP TABLE ch7_query;
```
If the row count differs from expectations, first check time boundaries and the count before the
join. Add predicates one at a time to locate where rows are excluded.
---
title: "7.6 Indexes and Performance"
url: https://docs.machbase.com/dbms/log-table-usage/index-performance/
language: en
kind: page
---
# 7.6 Indexes and Performance
If a query does not become faster after adding an index, first check whether it can use that index.
Indexes reduce read cost but increase ingestion, storage, and background processing costs. Start
with representative predicates rather than creating an index on every column.
## Choosing Indexes
| Query predicate | Index to consider | What to check |
|---|---|---|
| Values and ranges for numeric, DATETIME, and other supported types | LSM | Supported types and the actual execution plan |
| Words and token patterns in VARCHAR or TEXT | KEYWORD | Uses SEARCH/ESEARCH; result semantics differ from LIKE |
| Repeated-value analysis on supported types | BITMAP | Value distribution, encoding, ingestion cost, and storage cost |
For LOG `_arrival_time` ranges, use the built-in time access path first. Do not add an index for the
same purpose by habit. Check supported types and properties in
[INDEX Syntax](/dbms/reference/sql/syntax/index-syntax/). Do not directly apply conventional RDBMS
composite-index designs.
## Comparing Before and After Index Creation
```sql
CREATE LOG TABLE ch7_index (
event_id INTEGER,
event_time DATETIME,
severity SHORT,
message VARCHAR(256)
);
INSERT INTO ch7_index VALUES (
1, TO_DATE('2026-01-01 10:00:00', 'YYYY-MM-DD HH24:MI:SS'), 1, 'service started');
INSERT INTO ch7_index VALUES (
2, TO_DATE('2026-01-01 10:01:00', 'YYYY-MM-DD HH24:MI:SS'), 3, 'database timeout');
INSERT INTO ch7_index VALUES (
3, TO_DATE('2026-01-01 10:02:00', 'YYYY-MM-DD HH24:MI:SS'), 3, 'connection timeout');
EXPLAIN SELECT event_id FROM ch7_index WHERE severity = 3;
CREATE INDEX ch7_index_time ON ch7_index(event_time) INDEX_TYPE LSM;
CREATE INDEX ch7_index_message ON ch7_index(message) INDEX_TYPE KEYWORD;
CREATE INDEX ch7_index_severity ON ch7_index(severity)
INDEX_TYPE BITMAP BITMAP_ENCODE = RANGE;
EXEC TABLE_FLUSH(ch7_index);
EXEC INDEX_FLUSH(ch7_index);
EXPLAIN SELECT event_id FROM ch7_index WHERE severity = 3;
EXPLAIN SELECT event_id FROM ch7_index WHERE message SEARCH 'timeout';
SELECT event_id FROM ch7_index
WHERE message SEARCH 'timeout'
ORDER BY event_id;
SHOW INDEXES;
```
The search returns rows 2 and 3. Compare the execution-plan access paths for the value predicate and
SEARCH, and use SHOW INDEXES to check the created names. These three rows illustrate behavior; they
are not a performance benchmark.
## Data and Index Synchronization
`TABLE_FLUSH` flushes table data, whereas `INDEX_FLUSH` waits for index building to progress. Use
them separately as shown when comparing plans and timings before and after index creation.
An index can exist before all ingested data has been indexed. Build lag can affect search cost.
However, calling both commands for every inserted row reduces the benefit of batch ingestion. In
production, monitor ingestion rate and background processing rate together, and synchronize only at
required points.
Do not reinsert data simply because indexing is delayed. Check the source row count and index state
separately before retrying to avoid duplicates.
## Performance Measurement Criteria
Compare with the same data volume, predicate values, and concurrent ingestion load. Record repeated
query timings, ingestion throughput, index space, and build lag, rather than a single execution
time. LIKE and REGEXP evaluate predicates against raw strings; narrowing the time range first
reduces the candidates. A KEYWORD index does not turn LIKE into SEARCH.
```sql
DROP INDEX ch7_index_severity;
DROP INDEX ch7_index_message;
DROP INDEX ch7_index_time;
DROP TABLE ch7_index;
```
If the plan is difficult to interpret, compare the query and EXPLAIN output together. The diagnostic
sequence in [Index Tuning](/dbms/performance-tuning/index-tuning/) helps identify the next checks.
---
title: "7.7 Operations and Data Lifecycle"
url: https://docs.machbase.com/dbms/log-table-usage/operations-lifecycle/
language: en
kind: page
---
# 7.7 Operations and Data Lifecycle
If ingestion works but disk usage keeps growing, check the deletion criteria. LOG removes older data
regions rather than selecting individual rows by business predicates. Check both the retention
period and the actual deletion boundary.
## Choosing a Deletion Method
| Goal | Command | Basis |
|---|---|---|
| Delete the oldest N rows | OLDEST n ROWS | Oldest ingested rows first |
| Keep only the latest N rows | EXCEPT n ROWS | Number of rows to retain |
| Keep only a recent period | EXCEPT n DAY, etc. | Current server time minus the period |
| Delete through a fixed timestamp | BEFORE datetime_expr | Includes the specified _arrival_time boundary |
| Delete all data | DELETE without a predicate, or TRUNCATE | All rows |
| Manage retention periodically | Retention Policy | Retention period and execution interval |
Caution: do not assume deletion can be undone. LOG data is outside TRANSACTION table ROLLBACK. In
production, verify backups and the actual target range before executing deletion.
## Comparing Deletion Methods
When commands run sequentially, earlier deletions affect later results. Here, copy the same three
rows into separate tables for comparison.
```sql
CREATE LOG TABLE ch7_lifecycle (event_id INTEGER);
CREATE LOG TABLE ch7_oldest (event_id INTEGER);
CREATE LOG TABLE ch7_keep (event_id INTEGER);
CREATE LOG TABLE ch7_before (event_id INTEGER);
INSERT INTO ch7_lifecycle(_arrival_time, event_id)
VALUES (TO_DATE('2026-01-01', 'YYYY-MM-DD'), 1);
INSERT INTO ch7_lifecycle(_arrival_time, event_id)
VALUES (TO_DATE('2026-01-02', 'YYYY-MM-DD'), 2);
INSERT INTO ch7_lifecycle(_arrival_time, event_id)
VALUES (TO_DATE('2026-01-03', 'YYYY-MM-DD'), 3);
INSERT INTO ch7_oldest(_arrival_time, event_id)
SELECT _arrival_time, event_id FROM ch7_lifecycle ORDER BY _arrival_time;
INSERT INTO ch7_keep(_arrival_time, event_id)
SELECT _arrival_time, event_id FROM ch7_lifecycle ORDER BY _arrival_time;
INSERT INTO ch7_before(_arrival_time, event_id)
SELECT _arrival_time, event_id FROM ch7_lifecycle ORDER BY _arrival_time;
SELECT COUNT(*) AS delete_candidates FROM ch7_before
WHERE _arrival_time <= TO_DATE('2026-01-02', 'YYYY-MM-DD');
DELETE FROM ch7_oldest OLDEST 1 ROWS;
DELETE FROM ch7_keep EXCEPT 1 ROWS;
DELETE FROM ch7_before BEFORE TO_DATE('2026-01-02', 'YYYY-MM-DD');
SELECT event_id FROM ch7_oldest ORDER BY event_id;
SELECT event_id FROM ch7_keep ORDER BY event_id;
SELECT event_id FROM ch7_before ORDER BY event_id;
```
The pre-deletion count is 2. The following events remain in each table.
| Table | Remaining event_id values |
|---|---|
| ch7_oldest | 2, 3 |
| ch7_keep | 3 |
| ch7_before | 3 |
The name BEFORE can be misleading. Current LOG deletion includes rows equal to the specified
timestamp. A preliminary count with `WHERE _arrival_time < boundary` can therefore differ from the
deletion target. Use `<=` as shown.
```sql
DELETE FROM ch7_lifecycle;
SELECT COUNT(*) AS remaining_rows FROM ch7_lifecycle;
DROP TABLE ch7_before;
DROP TABLE ch7_keep;
DROP TABLE ch7_oldest;
DROP TABLE ch7_lifecycle;
```
After DELETE without a predicate, the count is 0 and the table definition remains.
## Deletion by Relative Period
```sql
CREATE LOG TABLE ch7_period (event_id INTEGER);
INSERT INTO ch7_period(_arrival_time, event_id) VALUES (SYSDATE - 2d, 1);
INSERT INTO ch7_period(_arrival_time, event_id) VALUES (SYSDATE, 2);
DELETE FROM ch7_period EXCEPT 1 DAY;
SELECT event_id FROM ch7_period ORDER BY event_id;
DROP TABLE ch7_period;
```
If the steps from creation through querying run immediately, only row 2 remains. The reference is
the current server time, not the timestamp of the last ingested row. Account for time continuing to
pass even when ingestion stops.
## Retention Policy
The following validation example retains one day of data and runs every minute. These are not
production recommendations. Use an account authorized to create policies and assign them to tables.
```sql
CREATE LOG TABLE ch7_retention (event_id INTEGER);
INSERT INTO ch7_retention(_arrival_time, event_id) VALUES (SYSDATE - 2d, 1);
INSERT INTO ch7_retention(_arrival_time, event_id) VALUES (SYSDATE, 2);
CREATE RETENTION ch7_policy DURATION 1 DAY INTERVAL 1 MIN;
ALTER TABLE ch7_retention ADD RETENTION ch7_policy;
SELECT * FROM M$RETENTION WHERE POLICY_NAME = 'CH7_POLICY';
SELECT TABLE_NAME, POLICY_NAME, STATE, LAST_DELETED_TIME
FROM V$RETENTION_JOB WHERE TABLE_NAME = 'CH7_RETENTION';
SELECT event_id FROM ch7_retention ORDER BY event_id;
```
DURATION is the period to retain; INTERVAL is how often deletion runs. Both rows may be visible
immediately after assignment. After one interval plus processing time, rerun the query below and
verify that only row 2 remains. LAST_DELETED_TIME is the deletion cutoff, not the wall-clock
completion time.
```sql
SELECT TABLE_NAME, STATE, LAST_DELETED_TIME
FROM V$RETENTION_JOB WHERE TABLE_NAME = 'CH7_RETENTION';
SELECT event_id FROM ch7_retention ORDER BY event_id;
```
After the exercise, detach the policy before dropping the policy and table.
```sql
ALTER TABLE ch7_retention DROP RETENTION;
SELECT TABLE_NAME FROM V$RETENTION_JOB WHERE TABLE_NAME = 'CH7_RETENTION';
DROP RETENTION ch7_policy;
DROP TABLE ch7_retention;
```
After detachment, the job query returns 0 rows. Previously deleted rows are not restored. One policy
can be assigned to a table, and a policy in use must be detached before it can be dropped. For full
operational guidance, see
[Data Retention Policies](/dbms/operations-configuration-recovery/policy-data-retention/).
## Verifying Backups Before Deletion
Having a backup file alone does not establish recoverability. Query the period to be deleted through
Mount or in an isolated Restore environment. Define retention periods separately for source data,
backups, and independently aggregated data. For environment-specific commands, see
[Backup, Restore, and Mount](/dbms/operations-configuration-recovery/backup-restore-mount/).
## Data and Disk Space
Rows may disappear from queries before the operating system reclaims file space. Check ingestion
volume, the oldest remaining timestamp, index/storage cleanup state, and disk usage together. Do not
delete a wider period simply because disk usage does not decrease immediately.
If deletion results differ from expectations, first check the cutoff timestamp and assigned policy.
For data subject to retention obligations, stop further deletion and agree on the scope with the
responsible owner.
---
title: "7.8 Constraints, Errors, and Troubleshooting"
url: https://docs.machbase.com/dbms/log-table-usage/constraints-errors-troubleshooting/
language: en
kind: page
---
# 7.8 Constraints, Errors, and Troubleshooting
Repeating the same failed SQL can leave the cause unchanged while making the state harder to
understand. First distinguish unsupported operations from invalid input or object-state issues. This
section narrows down checks by symptom.
## Feature Support
| Request | LOG support | Alternative |
|---|---|---|
| General UPDATE | Unsupported | Design correction events or choose a mutable table |
| DELETE WHERE with arbitrary predicates | Unsupported | BEFORE, OLDEST, EXCEPT, or a different model |
| PRIMARY KEY/UNIQUE constraints | Unsupported | Handle duplicates during collection or choose another table |
| Value/range indexes | LSM supported | Choose according to type and predicate |
| Word search | KEYWORD supported | Create on VARCHAR/TEXT, then use SEARCH/ESEARCH |
| BITMAP analytical indexes | Subject to supported conditions | Check type, encoding, and value distribution |
| ORDER BY/GROUP BY on TEXT itself | Unsupported | Use separate code, severity, or time columns |
## Diagnosis by Symptom
| Symptom | First check | Action |
|---|---|---|
| Stored arrival time differs from the explicit value | Previous timestamp and TIME_INVERSION_MODE | Check adjustment; retain event time separately |
| SEARCH reports an index error | KEYWORD index on the column | Check type, table, and index name |
| A word exists but is not found | SEARCH tokens versus LIKE substrings | Compare both results on one raw row |
| Query remains slow after index creation | Plan, build state, and time range | Check indexing progress, then measure representative load |
| Column length change fails | Existing type and new length | Use VARCHAR expansion only; check the maximum |
| MINMAX change fails | Whether the type is variable-length | Target only supported fixed-length LOG columns |
| NOT NULL change fails | Existing NULL rows | Distinguish existing-data checks from NOCHECK semantics |
| An event appears multiple times | Source ID, retries, and file reprocessing | Review retransmission policy; arbitrary row deletion is unavailable |
| DDL reports a resource-in-use error | Conflicts with ingestion or queries | Reschedule operations and check again |
Check server settings and the index list as follows.
```sql
SELECT NAME, VALUE FROM V$PROPERTY
WHERE NAME IN ('DISK_COLUMNAR_TABLE_TIME_INVERSION_MODE', 'TABLE_SCAN_DIRECTION');
SHOW INDEXES;
```
Checking a setting and changing it are separate operations. Do not change production-wide settings
before identifying the cause.
## Reproducing and Resolving Errors
```sql
CREATE LOG TABLE ch7_error (event_id INTEGER, message TEXT);
INSERT INTO ch7_error VALUES (1, 'connection timeout');
SELECT event_id, message FROM ch7_error;
```
The query returns one row. Each statement below intentionally fails. Run only the statement you want
to verify, separately from the normal exercise.
```sql
-- No KEYWORD index exists.
SELECT event_id FROM ch7_error WHERE message SEARCH 'timeout';
-- Sorting and grouping TEXT itself are unsupported.
SELECT message FROM ch7_error ORDER BY message;
SELECT message, COUNT(*) FROM ch7_error GROUP BY message;
-- LOG does not support general UPDATE or predicate-based DELETE.
UPDATE ch7_error SET message = 'fixed' WHERE event_id = 1;
DELETE FROM ch7_error WHERE event_id = 1;
-- This statement does not convert TEXT to VARCHAR.
ALTER TABLE ch7_error MODIFY COLUMN (message VARCHAR(4096));
```
Now create the index and check the supported search path.
```sql
CREATE INDEX ch7_error_msg ON ch7_error(message) INDEX_TYPE KEYWORD;
EXEC TABLE_FLUSH(ch7_error);
EXEC INDEX_FLUSH(ch7_error);
SELECT event_id, message FROM ch7_error
WHERE message SEARCH 'timeout'
ORDER BY event_id;
DROP TABLE ch7_error;
```
Row 1 should be returned. Adding an index does not enable TEXT sorting or LOG UPDATE.
## Diagnosing Retention Policies
If expired rows remain, check the assigned policy, execution interval, LAST_DELETED_TIME, and actual
`_arrival_time` together. Do not conclude deletion failed based only on `event_time`. For exercises,
see [Operations and Data Lifecycle](../operations-lifecycle/).
If the issue persists, collect the server version and Edition, table DDL, failing SQL, complete
error message, and representative input values. Redact passwords and sensitive logs before sharing.
A small reproducer is more useful for diagnosis than sending the entire dataset.
---
title: "7.9 Patterns and Scenarios"
url: https://docs.machbase.com/dbms/log-table-usage/patterns-scenarios/
language: en
kind: page
---
# 7.9 Patterns and Scenarios
Real analysis combines time, host, severity, and message. After trying each feature separately, use
them together to answer which errors are increasing on which server. This exercise runs
independently of tables from earlier sections.
## Source Events and Current State
Append a new LOG row whenever an event occurs. Mutable reference data, such as a device's current
name or location, can be kept in LOOKUP. If historical state is required, retain the values in the
event or design separate history.
The same approach applies to security events and job tracking. Consider TAG first for measurement
aggregation by sensor name, or TRANSACTION for business-row updates and transactions.
## Creating Logs and Indexes
Explicit arrival timestamps make time predicates comparable regardless of the execution date. This
exercise inserts in ascending order into an empty table. For ordinary collection, keep the source
event time in a separate DATETIME column.
```sql
CREATE LOG TABLE ch7_app (
event_id INTEGER,
host VARCHAR(32),
level VARCHAR(16),
message TEXT
);
CREATE INDEX ch7_app_message ON ch7_app(message) INDEX_TYPE KEYWORD;
INSERT INTO ch7_app(_arrival_time, event_id, host, level, message)
VALUES (TO_DATE('2026-01-01 10:00:00', 'YYYY-MM-DD HH24:MI:SS'),
1, 'web-01', 'INFO', 'service started');
INSERT INTO ch7_app(_arrival_time, event_id, host, level, message)
VALUES (TO_DATE('2026-01-01 10:10:00', 'YYYY-MM-DD HH24:MI:SS'),
2, 'web-01', 'WARN', 'slow response');
INSERT INTO ch7_app(_arrival_time, event_id, host, level, message)
VALUES (TO_DATE('2026-01-01 10:20:00', 'YYYY-MM-DD HH24:MI:SS'),
3, 'web-02', 'ERROR', 'database timeout');
INSERT INTO ch7_app(_arrival_time, event_id, host, level, message)
VALUES (TO_DATE('2026-01-01 10:30:00', 'YYYY-MM-DD HH24:MI:SS'),
4, 'web-02', 'ERROR', 'connection refused');
INSERT INTO ch7_app(_arrival_time, event_id, host, level, message)
VALUES (TO_DATE('2026-01-01 11:00:00', 'YYYY-MM-DD HH24:MI:SS'),
5, 'web-01', 'INFO', 'normal service');
EXEC TABLE_FLUSH(ch7_app);
EXEC INDEX_FLUSH(ch7_app);
SELECT COUNT(*) AS received_rows FROM ch7_app;
```
The inserted row count is 5. Before repeating the exercise, confirm that the final DROP ran. Simply
resending data can create duplicate rows.
## Querying Errors and Time Ranges
```sql
SELECT event_id, host, message FROM ch7_app
WHERE _arrival_time >= TO_DATE('2026-01-01 10:00:00', 'YYYY-MM-DD HH24:MI:SS')
AND _arrival_time < TO_DATE('2026-01-01 11:00:00', 'YYYY-MM-DD HH24:MI:SS')
AND message SEARCH 'timeout'
ORDER BY event_id;
SELECT event_id, host, level, message FROM ch7_app
WHERE _arrival_time >= TO_DATE('2026-01-01 10:00:00', 'YYYY-MM-DD HH24:MI:SS')
AND _arrival_time < TO_DATE('2026-01-01 11:00:00', 'YYYY-MM-DD HH24:MI:SS')
AND level = 'ERROR'
ORDER BY event_id;
```
The first query selects event 3 on web-02; the second selects events 3 and 4. This workflow starts
with a word search, then examines other errors for the same host and period. Expand only the needed
interval instead of removing the time predicate entirely.
## Aggregating by Hour and Severity
```sql
SELECT TO_CHAR(_arrival_time, 'YYYY-MM-DD HH24') AS event_hour,
level, COUNT(*) AS event_count
FROM ch7_app
WHERE _arrival_time >= TO_DATE('2026-01-01 10:00:00', 'YYYY-MM-DD HH24:MI:SS')
AND _arrival_time < TO_DATE('2026-01-01 12:00:00', 'YYYY-MM-DD HH24:MI:SS')
GROUP BY TO_CHAR(_arrival_time, 'YYYY-MM-DD HH24'), level
ORDER BY event_hour, level;
```
| event_hour | level | event_count |
|---|---|---|
| 2026-01-01 10 | ERROR | 2 |
| 2026-01-01 10 | INFO | 1 |
| 2026-01-01 10 | WARN | 1 |
| 2026-01-01 11 | INFO | 1 |
This query aggregates source LOG rows. It does not automatically use TAG ROLLUP. As data grows,
check the query range and execution plan, then separately assess whether preaggregation is needed.
A common mistake is applying `DURATION 1 HOUR` to data with fixed timestamps. That predicate uses
the current time and may exclude the samples on another execution date. Distinguish recent-log
queries in production from reproducible fixed-time queries.
## Collection and Retention Management
Continue with [Append Ingestion](../data-input-mutation/) for continuous collection. For long-term
operation, also define a [Retention Policy](../operations-lifecycle/). Source logs, backups, and
separate aggregates can require different retention periods.
```sql
DROP TABLE ch7_app;
```
Once the results match, substitute fields and messages from a few actual logs. Testing whether a
small sample answers the same questions makes issues easier to identify before migrating the full
collection pipeline.
---
title: "7.10 _arrival_time Time Model"
url: https://docs.machbase.com/dbms/log-table-usage/arrival-time-model/
language: en
kind: page
---
# 7.10 _arrival_time Time Model
A source log may contain yesterday's timestamp while a query treats it as data collected today.
Mixing event time and collection time can make valid ingestion look like missing data.
Distinguishing them is fundamental to LOG queries and retention policies.
## Event Time and Arrival Time
A user-defined DATETIME column such as `event_time` answers when the event occurred. The
automatically created `_arrival_time` is the basis for LOG time-range access and retention deletion.
If omitted, it uses the server time. However, explicit input and out-of-order adjustment mean that
it does not always represent the actual network reception time.
DATETIME represents values in nanoseconds. This does not mean the server clock measures every
timestamp with nanosecond precision. Once stored, a timestamp cannot be corrected with UPDATE on a
LOG table.
## Querying Late Events
This example inserts explicit arrival timestamps in ascending order into an empty table.
```sql
CREATE LOG TABLE ch7_time (
event_id INTEGER,
event_time DATETIME,
message VARCHAR(64)
);
INSERT INTO ch7_time(_arrival_time, event_id, event_time, message)
VALUES (TO_DATE('2026-01-02 10:00:00', 'YYYY-MM-DD HH24:MI:SS'), 1,
TO_DATE('2026-01-02 09:59:00', 'YYYY-MM-DD HH24:MI:SS'), 'normal arrival');
INSERT INTO ch7_time(_arrival_time, event_id, event_time, message)
VALUES (TO_DATE('2026-01-02 10:01:00', 'YYYY-MM-DD HH24:MI:SS'), 2,
TO_DATE('2026-01-01 23:00:00', 'YYYY-MM-DD HH24:MI:SS'), 'delayed arrival');
SELECT event_id
FROM ch7_time
WHERE event_time >= TO_DATE('2026-01-01', 'YYYY-MM-DD')
AND event_time < TO_DATE('2026-01-02', 'YYYY-MM-DD')
ORDER BY event_id;
SELECT event_id
FROM ch7_time
DURATION FROM TO_DATE('2026-01-02 10:00:00', 'YYYY-MM-DD HH24:MI:SS')
TO TO_DATE('2026-01-02 10:01:00', 'YYYY-MM-DD HH24:MI:SS');
```
The event-time query selects only event 2; the arrival-time query selects both events 1 and 2. There
is no need to alter the event timestamp of a late arrival.
## Out-of-Order Timestamps and Adjustment
`DISK_COLUMNAR_TABLE_TIME_INVERSION_MODE` controls how a timestamp earlier than the previous arrival
timestamp is handled. The current Standard implementation behaves as follows.
| Value | Handling of out-of-order input |
|---|---|
| 1 (default) | Adjust to 1 ns after the previously stored `_arrival_time` |
| 0 | Reject the insert with a time-inversion error |
Equal timestamps are not out of order, so this rule does not guarantee a unique timestamp for every
row. To guarantee ordering when timestamps are equal, add another key, such as an event number, to
ORDER BY.
The following optional exercise uses the same table. Check the setting first and run it only in a
validation environment where the value is 1. Do not change a production server setting for this
example.
```sql
SELECT NAME, VALUE FROM V$PROPERTY
WHERE NAME = 'DISK_COLUMNAR_TABLE_TIME_INVERSION_MODE';
```
```sql
-- Run with setting 1. Setting 0 is expected to reject the insert.
INSERT INTO ch7_time(_arrival_time, event_id, event_time, message)
VALUES (TO_DATE('2026-01-02 09:00:00', 'YYYY-MM-DD HH24:MI:SS'), 3,
TO_DATE('2026-01-02 08:59:00', 'YYYY-MM-DD HH24:MI:SS'), 'inverted arrival');
SELECT event_id,
TO_CHAR(_arrival_time, 'YYYY-MM-DD HH24:MI:SS mmm:uuu:nnn') AS stored_time
FROM ch7_time
ORDER BY event_id;
```
With setting 1, event 3 is stored at `2026-01-02 10:01:00 000:000:001`, not the supplied 09:00
timestamp. The event time remains unchanged. Allowing time inversion therefore does not mean
preserving every historical timestamp as supplied.
## Data Migration Considerations
To preserve source `_arrival_time` values, normally insert into an empty target in ascending order.
Even sorted input can be adjusted or rejected if the target already contains newer rows or
concurrent ingestion intervenes. Do not mix migration and ordinary real-time collection in the same
table.
`DURATION` uses `_arrival_time`, not `event_time`. If the range differs even after matching time
zones and date formats, check which time column the query uses. For boundaries and output order,
continue to [Query Examples](../query-analysis/).
```sql
DROP TABLE ch7_time;
```
When reporting a time-related issue, provide the source timestamp, stored timestamp, and setting
value. Comparing all three helps distinguish conversion from adjustment.
---
title: "7.11 Text Search and KEYWORD Index"
url: https://docs.machbase.com/dbms/log-table-usage/text-search-keyword-index/
language: en
kind: page
---
# 7.11 Text Search and KEYWORD Index
SEARCH and LIKE can return different results even when messages contain the same characters. SEARCH
finds indexed words; LIKE applies a pattern to the raw string. Before comparing performance,
establish which rows each query should find.
## Prepare Search Data
```sql
CREATE LOG TABLE ch7_search (
event_id INTEGER,
message TEXT
);
CREATE INDEX ch7_search_msg ON ch7_search(message) INDEX_TYPE KEYWORD;
INSERT INTO ch7_search VALUES (1, 'ERROR connection timeout');
INSERT INTO ch7_search VALUES (2, 'connection slowly refused');
INSERT INTO ch7_search VALUES (3, 'pretimeout marker');
INSERT INTO ch7_search VALUES (4, 'normal service');
INSERT INTO ch7_search VALUES (5, NULL);
INSERT INTO ch7_search VALUES (6, '대한민국 연결 오류');
INSERT INTO ch7_search VALUES (7, 'ERR-1001 network');
INSERT INTO ch7_search VALUES (8, 'timeout refused connection');
EXEC TABLE_FLUSH(ch7_search);
EXEC INDEX_FLUSH(ch7_search);
```
Compare results by the sample `event_id` values. The message column is TEXT and is not used for
sorting. The same KEYWORD search is available for LOG VARCHAR columns.
## SEARCH and NOT SEARCH
```sql
SELECT event_id FROM ch7_search WHERE message SEARCH 'timeout' ORDER BY event_id;
SELECT event_id FROM ch7_search WHERE message SEARCH 'connection refused' ORDER BY event_id;
SELECT event_id FROM ch7_search
WHERE message SEARCH 'connection' AND message SEARCH 'refused'
ORDER BY event_id;
SELECT event_id FROM ch7_search WHERE message NOT SEARCH 'timeout' ORDER BY event_id;
```
| Predicate | Selected event_id values | Reason |
|---|---|---|
| SEARCH 'timeout' | 1, 8 | pretimeout is a separate word |
| SEARCH 'connection refused' | 2, 8 | Both words are present |
| Two SEARCH predicates combined with AND | 2, 8 | Both words are checked in the same message |
| NOT SEARCH 'timeout' | 2, 3, 4, 6, 7 | The word is absent; NULL rows are excluded |
Multiword SEARCH is not phrase search: it does not guarantee word order or adjacency. Row 2 has an
intervening word and row 8 reverses the order, but both match. To SEARCH another column, that column
also needs the corresponding index.
Default tokenization normalizes ordinary ASCII words to lowercase. For example, `ERROR` in row 1
also matches `SEARCH 'error'`. Do not generalize this behavior to language-specific case handling
for all Unicode characters.
### Korean Tokenization
```sql
SELECT event_id FROM ch7_search WHERE message SEARCH '대한' ORDER BY event_id;
SELECT event_id FROM ch7_search WHERE message SEARCH '연결' ORDER BY event_id;
```
Both queries select row 6. In the default mode, `대한민국` is indexed as overlapping 2-grams such as
`대한`, `한민`, and `민국`. This is not morphological or semantic analysis. Test real samples because
spaces, punctuation, single characters, and mixed English/Korean values can affect token boundaries.
Index options such as MODE can also change tokenization.
## ESEARCH Extended Search
```sql
SELECT event_id FROM ch7_search WHERE message ESEARCH 'time%' ORDER BY event_id;
SELECT event_id FROM ch7_search WHERE message ESEARCH '%time%' ORDER BY event_id;
SELECT event_id FROM ch7_search WHERE message ESEARCH 'err%' ORDER BY event_id;
```
| Pattern | Selected event_id values | Meaning |
|---|---|---|
| time% | 1, 8 | Words starting with time |
| %time% | 1, 3, 8 | Words containing time |
| err% | 1, 7 | Words starting with err, such as error or err |
`time%` does not match time in the middle of a word. ESEARCH is also not equivalent to applying LIKE
to the entire raw string. Do not transfer raw patterns spanning spaces or punctuation directly to
ESEARCH. Make complex conditions explicit by combining separate SEARCH/ESEARCH predicates with AND
or OR.
ESEARCH is case-insensitive for ASCII in the current comparison path. Broader keyword matches
increase search cost, so it is not always faster than LIKE. This example uses ASCII keyword
patterns.
`NOT ESEARCH` syntax is unsupported. Substituting `NOT SEARCH` or `NOT LIKE` changes the search
semantics. Redefine which rows to exclude and check NULL handling.
## LIKE and NOT LIKE
```sql
SELECT event_id FROM ch7_search WHERE message LIKE '%TIMEOUT%' ORDER BY event_id;
SELECT event_id FROM ch7_search WHERE message LIKE 'ERR-____' ORDER BY event_id;
SELECT event_id FROM ch7_search WHERE message NOT LIKE '%timeout%' ORDER BY event_id;
```
The first query selects rows 1, 3, and 8; the second returns 0 rows; the third selects 2, 4, 6, and
7. Row 7 has text after `ERR-1001`, so it does not match the whole-string pattern `ERR-____`. `%`
matches zero or more characters; `_` matches one character. Check backslash escaping rules when
searching for literal `%`, `_`, or backslashes.
LIKE is also case-insensitive for ASCII in the current comparison path. It does not use a KEYWORD
index, but other WHERE predicates or a time range can reduce the rows tested. It is therefore also
inaccurate to say that LIKE always scans the entire table.
## REGEXP and REGEXP_LIKE
```sql
SELECT event_id FROM ch7_search
WHERE message REGEXP '^ERR-[0-9]+'
ORDER BY event_id;
SELECT event_id FROM ch7_search
WHERE message NOT REGEXP 'timeout'
ORDER BY event_id;
```
The first query selects row 7; the second selects 2, 4, 6, and 7. REGEXP checks for a matching
substring. Specify `^` or `$` to constrain a match to the start or end of the string.
Use `REGEXP_LIKE` for the function form. Its input currently must be VARCHAR, and its pattern and
options must be constant VARCHAR values. Passing the preceding TEXT column directly causes a type
error, so prepare a separate sample.
```sql
CREATE LOG TABLE ch7_regexp_fn (event_id INTEGER, message VARCHAR(200));
INSERT INTO ch7_regexp_fn VALUES (1, 'ERROR connection timeout');
INSERT INTO ch7_regexp_fn VALUES (5, NULL);
SELECT event_id,
REGEXP_LIKE(message, 'error') AS case_sensitive,
REGEXP_LIKE(message, 'error', 'i') AS case_insensitive
FROM ch7_regexp_fn
WHERE event_id IN (1, 5)
ORDER BY event_id;
```
Row 1 returns 0 and 1, respectively. Row 5, whose message is NULL, returns NULL for both results.
Regular expressions are case-sensitive by default; the `i` option makes comparison case-insensitive.
Use `c` for explicitly case-sensitive comparison.
Regular expressions are not processed directly by KEYWORD indexes. Time predicates or SEARCH can
narrow the candidates, but a regular expression cannot recover a row excluded by the preceding
predicate.
## TEXT Limitations and Search Performance
TEXT can hold raw content up to 64 MiB, but ORDER BY and GROUP BY on TEXT itself are unsupported.
Keep devices, error codes, and severity values for sorting and aggregation in separate columns.
Compare performance on the same data after checking index presence, build state, and query range.
```sql
DROP TABLE ch7_regexp_fn;
DROP TABLE ch7_search;
```
If search results differ, inspect one raw row alongside the pattern. Distinguishing word search from
substring search often resolves the discrepancy.
---
title: "7.12 Network Type Queries"
url: https://docs.machbase.com/dbms/log-table-usage/regex-network-query/
language: en
kind: page
---
# 7.12 Network Type Queries
Storing IP addresses as strings is readable, but can lead to the mistaken assumption that lexical
order matches address-range order. Use IPV4/IPV6 types for address comparison, and retain a separate
string only when the original notation is also required.
## Prepare Network Data
```sql
CREATE LOG TABLE ch7_network (
event_id INTEGER,
src_ip IPV4,
dst_ip IPV6,
dst_port INTEGER
);
INSERT INTO ch7_network VALUES (1, '192.0.2.1', '2001:db8::1', 80);
INSERT INTO ch7_network VALUES (2, '192.0.2.255', '2001:db8::2', 65535);
INSERT INTO ch7_network VALUES (3, '198.51.100.1', '2001:db8::3', 443);
INSERT INTO ch7_network VALUES (4, NULL, NULL, NULL);
SELECT event_id, src_ip, dst_ip, dst_port FROM ch7_network ORDER BY event_id;
```
The query returns four rows. Ports use INTEGER because USHORT reserves its maximum value, 65535, for
NULL and cannot represent the full port range.
## Equality and Range Queries
```sql
SELECT event_id FROM ch7_network
WHERE src_ip BETWEEN '192.0.2.1' AND '192.0.2.255'
ORDER BY event_id;
SELECT event_id FROM ch7_network
WHERE src_ip IN ('192.0.2.1', '198.51.100.1')
ORDER BY event_id;
SELECT event_id FROM ch7_network
WHERE dst_ip = '2001:db8::2'
ORDER BY event_id;
SELECT event_id FROM ch7_network
WHERE dst_ip BETWEEN '2001:db8::1' AND '2001:db8::2'
ORDER BY event_id;
SELECT event_id FROM ch7_network
WHERE src_ip IS NULL
ORDER BY event_id;
```
| Predicate | Selected event_id values |
|---|---|
| IPv4 BETWEEN | 1, 2 |
| IPv4 IN | 1, 3 |
| IPv6 equality | 2 |
| IPv6 BETWEEN | 1, 2 |
| IPv4 IS NULL | 4 |
BETWEEN includes both endpoint addresses. The IPv4 range here is explicitly bounded by two
addresses; it does not automatically apply CIDR semantics. Use `CONTAINED` below to test network
membership. Test NULL with `IS NULL`, not `= NULL`.
## Testing CIDR Network Membership
Use `CONTAINED` to test whether an address belongs to a network. Specify the network as
`address/prefix`. Both IPv4 and IPv6 are supported.
```sql
SELECT event_id FROM ch7_network
WHERE src_ip CONTAINED '192.0.2.0/24'
ORDER BY event_id;
SELECT event_id FROM ch7_network
WHERE dst_ip CONTAINED '2001:db8::/32'
ORDER BY event_id;
SELECT event_id FROM ch7_network
WHERE src_ip NOT CONTAINED '192.0.2.0/24'
ORDER BY event_id;
```
| Predicate | Selected event_id values |
|---|---|
| IPv4 `CONTAINED '192.0.2.0/24'` | 1, 2 |
| IPv6 `CONTAINED '2001:db8::/32'` | 1, 2, 3 |
| IPv4 `NOT CONTAINED '192.0.2.0/24'` | 3 |
The reversed form `'192.0.2.0/24' CONTAINS src_ip` has the same meaning. `CONTAINS` places the
network on the left and the address on the right; `CONTAINED` does the reverse.
Omitting the prefix, as in `CONTAINED '192.0.2.0'`, causes an error because the value cannot be
interpreted as a network. Row 4 has a NULL address and matches none of these predicates. Add a
separate `IS NULL` predicate when counting uncollected addresses.
## Address Formats and Preserving Source Notation
Use IPV4 when input contains only IPv4 addresses and IPV6 for IPv6 addresses. If both are handled,
define input-conversion and column-separation policies first and validate them with real samples. Do
not assume implicit conversion always normalizes source addresses as intended.
Pay attention to output strings. The same IPv6 address may be displayed differently from its
compressed source notation. If the original notation must be retained as evidence, store the typed
address and raw string in separate columns.
For large datasets, limit the `_arrival_time` range together with address predicates and inspect the
execution plan. For message regular expressions, see
[Text Search](../text-search-keyword-index/#regex) and the
[SQL Function Dictionary](/dbms/reference/sql/functions/).
```sql
DROP TABLE ch7_network;
```
If an address query returns unexpected results, compare the source string, input type, and both
range endpoints. This helps distinguish notation differences from actual address-range differences.
---
title: "8. TRANSACTION Table Usage"
url: https://docs.machbase.com/dbms/rdb-table-usage/
language: en
kind: section
---
# 8. TRANSACTION Table Usage
Appending source logs differs from changing order status, inventory, or device information.
State-changing workloads must also check how many rows changed, what rolls back after a partial
failure, and how concurrent updates from another connection behave.
TRANSACTION tables support these relational queries and modifications in Standard Edition only. This
chapter connects schema design, updates, transactions, concurrent access, and recovery while
checking results on small samples. Although SQLite is used internally for storage, Machbase SQL
defines the public syntax and support scope. Not every SQLite or other RDBMS feature is available
unchanged.
## Chapter Contents
| Section | What to check |
|---|---|
| [8.1 Overview and Use Criteria](./overview-use-criteria/) | Roles compared with LOG, TAG, and LOOKUP |
| [8.2 Table Structure and Schema](./table-structure-schema/) | Identifiers, business keys, types, and constraints |
| [8.3 Create, Alter, and Drop](./create-alter-drop/) | DDL and existing-data checks |
| [8.4 Data Ingestion and Modification](./data-input-mutation/) | Conditional updates/deletes, copying, and Append |
| [8.5 Queries and Analysis](./query-analysis/) | Filtering, sorting, aggregation, and JSON queries |
| [8.6 Indexes and Performance](./index-performance/) | Primary key, unique, composite, and JSON path indexes |
| [8.7 Operations and Data Lifecycle](./operations-lifecycle/) | Batch cleanup and operational checks |
| [8.8 Constraints, Errors, and Troubleshooting](./constraints-errors-troubleshooting/) | Symptom diagnosis and retry decisions |
| [8.9 Transactions](./transaction/) | Statement failures, ROLLBACK, and commit guarantees |
| [8.10 Locks, Conflicts, and Busy Timeout](./locking-conflict-timeout/) | Two-connection conflicts and snapshot retries |
| [8.11 JOIN and Relational Query Design](./join-relational-query/) | Rows added or excluded by joins |
| [8.12 Backup, Restore, and Mount](./backup-restore-mount/) | Verifying actual data at the backup point |
| [8.13 INSERT ON DUPLICATE KEY UPDATE](./insert-on-duplicate-key-update/) | Insert/update branches and duplicate handling |
For common automatic-numbering syntax, see
[AUTO_INCREMENT](/dbms/reference/sql/syntax/auto-increment-syntax/).
## Example Environment and Execution Scope
SQL exercises target a DBMS 8.7 Standard Edition validation environment. Each section creates and
cleans up `ch8_` objects independently. Table/index creation privileges are required; backup
exercises also require separate privileges and server paths.
Run BEGIN through COMMIT/ROLLBACK on the same connection. Follow the specified A/B sequence in
two-session exercises. Intentionally failing SQL is separated from normal flow. Complete cleanup SQL
before rerunning an exercise.
Caution: the TRANSACTION name does not mean all operations and all failure scenarios roll back
together. First check DDL, writes to other table types, and cross-table commit boundaries during
failures in [8.9 Transactions](./transaction/).
---
title: "8.1 Overview and Use Criteria"
url: https://docs.machbase.com/dbms/rdb-table-usage/overview-use-criteria/
language: en
kind: page
---
# 8.1 Overview and Use Criteria
Device measurements accumulate continuously, while inspection state and inventory quantities require
updates to existing values. Using one model for both can mix source-retention requirements with
state changes. Start with TRANSACTION for mutable business data and TAG/LOG for source time series.
## TRANSACTION Table Characteristics
TRANSACTION supports SELECT, INSERT, UPDATE, DELETE, PRIMARY KEY, UNIQUE INDEX, and secondary
indexes. It is Standard Edition only. The following three forms create the same table type.
| Syntax | Meaning |
|---|---|
| CREATE TABLE | Default TRANSACTION creation with the type omitted |
| CREATE TRANSACTION TABLE | Explicit type |
| CREATE TXN TABLE | Abbreviated type |
Use CREATE TRANSACTION TABLE in public documentation and operational scripts to make the type
explicit. CREATE RDB TABLE and CREATE TRX TABLE are unsupported. None of the three supported
creation forms is available in Cluster. Specify CREATE LOG TABLE explicitly for LOG.
## State Changes and Rollback
```sql
CREATE TRANSACTION TABLE ch8_overview (
item_id LONG PRIMARY KEY,
qty INTEGER NOT NULL
);
INSERT INTO ch8_overview VALUES (42, 10);
BEGIN;
UPDATE ch8_overview SET qty = qty - 3 WHERE item_id = 42 AND qty >= 3;
SELECT item_id, qty FROM ch8_overview;
ROLLBACK;
SELECT item_id, qty FROM ch8_overview;
DROP TABLE ch8_overview;
```
On the same connection, the query inside the transaction shows quantity 7; after ROLLBACK, it shows
10. `qty >= 3` is the business predicate that prevents a change when stock is insufficient.
Do not interpret an error-free UPDATE as business success. If no row matches, the affected row count
can be 0. The application must check that the expected one row was affected before deciding to
continue or roll back. This check matters more than simply substituting numbers in the SQL example.
## Comparison with Other Tables
| Main requirement | Table to consider first |
|---|---|
| Measurements by sensor name and ROLLUP | TAG |
| Immutable source logs/events | LOG |
| Small current reference datasets | LOOKUP |
| Relational DML and explicit transactions | TRANSACTION |
| In-memory state that may be lost on restart | VOLATILE |
TRANSACTION can hold equipment inspection state, business history, and separate summary results. For
bulk source collection, compare throughput and ingestion paths with TAG/LOG. TRANSACTION also
supports Append, but do not assume identical throughput or batch boundaries. See
[Ingestion Methods](../data-input-mutation/).
LOOKUP does not replace every TRANSACTION feature. If relational transactions are required in
Cluster, consider an architecture that includes a separate RDBMS.
## Design Criteria
Distinguish the row identifier from the business key used to prevent duplicates. An internal number
may use PRIMARY KEY, while a separate unique value such as an external-system code may need UNIQUE
INDEX. Automatic numbering does not eliminate duplicate business keys.
Next, define frequent WHERE predicates and business success criteria. Decide where transactions end
and what to check after errors so concurrent requests or connection failures have clear handling.
Read [Schema](../table-structure-schema/) together with [Transactions](../transaction/).
---
title: "8.2 Table Structure and Schema"
url: https://docs.machbase.com/dbms/rdb-table-usage/table-structure-schema/
language: en
kind: page
---
# 8.2 Table Structure and Schema
Automatic numbering does not eliminate all duplicates: the same external device can still be
inserted twice with different numbers. Schema design starts by distinguishing row identifiers from
keys that prevent business duplicates.
## Internal Identifiers and Business Keys
This example manages an internal number separately from an external device code.
```sql
CREATE TRANSACTION TABLE ch8_schema (
id LONG PRIMARY KEY AUTO_INCREMENT,
external_code VARCHAR(64) NOT NULL,
device_name VARCHAR(128) NOT NULL,
price DECIMAL(18,2),
state JSON,
updated_at DATETIME
);
CREATE UNIQUE INDEX ch8_schema_code ON ch8_schema(external_code);
INSERT INTO ch8_schema(external_code, device_name, price, state, updated_at)
VALUES ('ERP-01', 'Pump A', 19900.25, '{"status":"NORMAL"}',
TO_DATE('2026-01-01 10:00:00', 'YYYY-MM-DD HH24:MI:SS'));
SELECT external_code, device_name, price, state->'$.status' AS status
FROM ch8_schema;
```
One ERP-01 row is returned with price 19900.25 and state NORMAL. The server assigns id. Do not make
consecutive or gap-free numbering a business requirement. For retrieving assigned IDs, check your
SDK and [AUTO_INCREMENT](/dbms/reference/sql/syntax/auto-increment-syntax/).
## PRIMARY KEY and UNIQUE
A TRANSACTION table permits one single-column PRIMARY KEY. Specify PRIMARY KEY after the column
definition, or add it to an existing table with CREATE PRIMARY KEY INDEX. It cannot be created on
data containing NULL or duplicate values.
Use a composite UNIQUE INDEX to enforce uniqueness across multiple columns. Do not copy UNIQUE,
FOREIGN KEY, or table-level PRIMARY KEY syntax inside CREATE TABLE from another DBMS. Define
uniqueness with CREATE UNIQUE INDEX after table creation.
UNIQUE INDEX keys containing NULL are not considered duplicates of other NULL-containing keys. If a
code must exist and be unique, also declare NOT NULL as in the example. Empty strings are another
potential pitfall. Check Machbase empty-string/NULL behavior through the actual input path and
validate required codes during collection.
The following optional exercise checks a UNIQUE violation. Run it separately from successful inserts.
```sql
-- Expected failure: duplicate external_code
INSERT INTO ch8_schema(external_code, device_name)
VALUES ('ERP-01', 'Duplicate Pump');
```
After failure, ERP-01 should still have one row. To update on duplicates, use the separate
[UPSERT](../insert-on-duplicate-key-update/) rules.
## Choosing Data Types
| Value | Type choice | What to check |
|---|---|---|
| Identifiers and quantities | SHORT, INTEGER, LONG, and supported unsigned types | Range and reserved NULL values |
| Measurements and approximate values | FLOAT, DOUBLE | Floating-point rounding |
| Money and exact decimals | DECIMAL(M,D), NUMERIC, and other aliases | Precision, scale, and input conversion |
| Codes and names | VARCHAR(n) | Byte length, not character count |
| Long text and binary data | TEXT/CLOB, BINARY/BLOB | Storage support versus sorting, function, and index support |
| Event/change timestamps | DATETIME | Source time zone and conversion format |
| Network addresses | IPV4, IPV6 | Address format and comparison semantics |
| Additional attributes | JSON | Frequently searched paths and their types |
| Fixed-length numeric collections | Numeric ARRAY | Element type/length, whole-array NULL, and NULL elements |
Check complete ranges in the [Data Type Dictionary](/dbms/reference/sql/types/) and monetary values
in [DECIMAL](/dbms/reference/sql/types/decimal-numeric-fixed-point/). Do not use DOUBLE for every
monetary column simply because an example does.
## Constraints and Input Validation
TRANSACTION requires at least one user column. LOG automatic arrival timestamps and TAG
METADATA/BASETIME/BASEDISTANCE are unavailable. Do not assume foreign keys automatically enforce
referential integrity; include required relationship validation in the application and data checks.
The example UNIQUE INDEX does not validate business rules for device names or prices. Define
required values, allowed states, and quantity ranges separately.
```sql
SELECT COUNT(*) AS device_count FROM ch8_schema;
DROP TABLE ch8_schema;
```
The count before cleanup is 1. Continue with [Create, Alter, and Drop](../create-alter-drop/) for
schema changes and [Index Design](../index-performance/) for query access paths.
---
title: "8.3 Create, Alter, and Drop"
url: https://docs.machbase.com/dbms/rdb-table-usage/create-alter-drop/
language: en
kind: page
---
# 8.3 Create, Alter, and Drop
Schema changes require checking existing data and ingestion programs as well as command success. Old
SQL after a rename or dropping an indexed column before its index can cause errors. This section
checks before/after behavior with sample data present.
## Creating a Table
```sql
CREATE TRANSACTION TABLE ch8_ddl (
id LONG,
code VARCHAR(32),
qty INTEGER
);
INSERT INTO ch8_ddl VALUES (1, 'P-01', 10);
```
## Creating Keys and Indexes
```sql
CREATE PRIMARY KEY INDEX ch8_ddl_pk ON ch8_ddl(id);
CREATE UNIQUE INDEX ch8_ddl_code ON ch8_ddl(code);
CREATE INDEX ch8_ddl_qty ON ch8_ddl(qty);
SHOW INDEX ch8_ddl_code;
```
id is the single PRIMARY KEY; code is a separate business key. Creation can fail if existing data
contains duplicates or NULL in the primary key column. Define composite uniqueness with CREATE
UNIQUE INDEX. UNIQUE constraint syntax inside CREATE TABLE is unsupported.
For automatic numbering, specify `LONG PRIMARY KEY AUTO_INCREMENT` at creation. This section's id
values are explicit, not automatic. Do not apply both creation patterns to the same object. See the
separate [AUTO_INCREMENT](/dbms/reference/sql/syntax/auto-increment-syntax/) exercise.
## Adding Columns and Defaults
```sql
ALTER TABLE ch8_ddl ADD COLUMN (label VARCHAR(64));
ALTER TABLE ch8_ddl ADD COLUMN (status VARCHAR(16) DEFAULT 'NEW');
ALTER TABLE ch8_ddl ADD COLUMN (limits DECIMAL(12)[2] DEFAULT [10, 20]);
SELECT id, label, status, limits FROM ch8_ddl ORDER BY id;
```
In row 1, label is NULL, status is NEW, and limits is [10, 20]. Adding an ARRAY column without
DEFAULT gives existing rows a whole-array NULL.
DEFAULT support differs between ADD COLUMN and CREATE TABLE. ADD COLUMN accepts a value compatible
with the type, as above. In CREATE TABLE column definitions, only `DEFAULT SYSDATE` on DATETIME is
supported. Other types and values are rejected with `ERR-02346` and `ERR-02347`, respectively. If an
initial default is needed, first create the table and add the column with ADD COLUMN, or specify the
value in INSERT. Distinguish a whole-array NULL from NULL in individual array elements.
Enclose column definitions in parentheses; this syntax differs from other DBMS ALTER TABLE forms.
TRANSACTION does not support changing length or type with MODIFY COLUMN. Prepare a separate
migration to a new schema when needed.
## Column Changes and Dependent Objects
```sql
DROP INDEX ch8_ddl_qty;
ALTER TABLE ch8_ddl DROP COLUMN (qty);
ALTER TABLE ch8_ddl DROP COLUMN (label);
ALTER TABLE ch8_ddl DROP COLUMN (limits);
ALTER TABLE ch8_ddl RENAME COLUMN code TO product_code;
SELECT id, product_code, status FROM ch8_ddl;
SHOW INDEX ch8_ddl_code;
```
Existing row 1 retains P-01 and NEW, and the business-key index remains. Check PRIMARY KEY, UNIQUE,
ordinary, and JSON path indexes before changing referenced columns. The last user column cannot be
dropped.
Renaming or dropping a table/column can be rejected if a VIEW references it. Application SQL and
prepared statements are also affected. After DDL, check whether existing prepared statements must be
prepared again rather than blindly reusing them.
```sql
ALTER TABLE ch8_ddl RENAME TO ch8_product;
SELECT id, product_code, status FROM ch8_product;
```
After renaming the table, query ch8_product.
## Deleting All Rows and Dropping Tables
```sql
BEGIN;
TRUNCATE TABLE ch8_product;
SELECT COUNT(*) AS during_delete FROM ch8_product;
ROLLBACK;
SELECT COUNT(*) AS after_rollback FROM ch8_product;
DROP TABLE ch8_product;
```
The counts are 0 and 1, respectively. Current TRANSACTION TRUNCATE is implemented as deletion of all
rows and can be rolled back inside an explicit transaction. Do not generalize this to LOG/TAG
TRUNCATE. The final DROP removes data, definition, and related indexes.
## Operational DDL Considerations
Distinguish this TRUNCATE behavior from schema operations such as CREATE, ALTER, and DROP. Do not
assume schema changes inside BEGIN can later be rolled back. Active transactions or open cursors on
the same table can block DDL; close result sets and finish business transactions first.
ADD/DROP COLUMN modify both the catalog and separate storage files. If the server stops during the
operation, do not repeat the DDL before restart recovery finishes. After recovery, check DESC,
representative SELECT/INSERT operations, indexes, views, and server logs. Do not recover by moving,
editing, or deleting internal storage files directly.
If the schema differs from expectations, review the original DDL, execution sequence, and first
error together. This is more useful than examining only the last error.
---
title: "8.4 Data Input and Mutation"
url: https://docs.machbase.com/dbms/rdb-table-usage/data-input-mutation/
language: en
kind: page
---
# 8.4 Data Input and Mutation
A successful UPDATE response does not necessarily mean an order reached the intended state. No
matching row can yield 0 affected rows without an error. Check the target before modification,
affected row count, and resulting values together.
## Conditional UPDATE and DELETE
```sql
CREATE TRANSACTION TABLE ch8_mutation (
order_id LONG PRIMARY KEY,
amount DECIMAL(18,2),
status VARCHAR(16),
ordered DATETIME
);
INSERT INTO ch8_mutation VALUES (
1001, 19900.25, 'PENDING', TO_DATE('2026-01-01', 'YYYY-MM-DD'));
INSERT INTO ch8_mutation VALUES (
1002, 29900.50, 'CANCELLED', TO_DATE('2026-01-02', 'YYYY-MM-DD'));
BEGIN;
UPDATE ch8_mutation SET status = 'SHIPPED'
WHERE order_id = 1001 AND status = 'PENDING';
SELECT order_id, status FROM ch8_mutation ORDER BY order_id;
COMMIT;
```
1001 is SHIPPED and 1002 is CANCELLED. Repeating the UPDATE affects 0 rows because the row is no
longer PENDING. Use the SDK's affected row count to distinguish success, already processed, and
missing targets according to business rules.
For the deletion below, also check the target count before executing it in a transaction.
```sql
BEGIN;
SELECT COUNT(*) AS delete_candidates FROM ch8_mutation WHERE status = 'CANCELLED';
DELETE FROM ch8_mutation WHERE status = 'CANCELLED';
SELECT order_id, amount, status FROM ch8_mutation ORDER BY order_id;
COMMIT;
```
One row is targeted; after deletion, only 1001 remains. UPDATE/DELETE without WHERE affect all rows.
In production, another session can change data between a preliminary SELECT and the actual
modification, so the preliminary count alone does not establish success.
## INSERT SELECT and Self-Reference
```sql
CREATE TRANSACTION TABLE ch8_archive (
order_id LONG PRIMARY KEY,
amount DECIMAL(18,2),
status VARCHAR(16),
ordered DATETIME
);
INSERT INTO ch8_archive(order_id, amount, status, ordered)
SELECT order_id, amount, status, ordered FROM ch8_mutation
WHERE ordered < TO_DATE('2026-02-01', 'YYYY-MM-DD');
INSERT INTO ch8_mutation(order_id, amount, status, ordered)
SELECT order_id + 10000, amount, status, ordered FROM ch8_mutation
WHERE order_id = 1001;
SELECT order_id FROM ch8_archive ORDER BY order_id;
SELECT order_id FROM ch8_mutation ORDER BY order_id;
```
The archive contains 1001; the source contains 1001 and 11001. Reading from and inserting into the
same table does not endlessly reinsert new rows in this example. Copying without changing keys can
still violate uniqueness. Match target column counts/types and partition copy ranges for safe
reruns.
Distinguish ordinary constraint errors from connection failures. Statement failure does not
automatically roll back the entire BEGIN transaction. If the commit response is lost, query again to
determine whether changes were applied. See [Transactions](../transaction/) for boundaries.
```sql
DROP TABLE ch8_archive;
DROP TABLE ch8_mutation;
```
## Bulk Ingestion and Batch Boundaries
TRANSACTION also supports the Append API. Old filenames or anchors containing reject/unsupported do
not indicate current lack of support. Choose public language APIs using
[SDK Feature Support](/dbms/development-tools-integration/sdk-support-scope/#support-scope-sdk-append).
| Ingestion method | What to check |
|---|---|
| SQL INSERT/prepared execution | Statement errors and explicit transaction boundaries |
| Driver batch | Actual transmission units, partial success, and autocommit |
| Append | SDK buffers/server batch boundaries, error callbacks, and return values |
| machloader | Mappings, rejected rows, and processed-range records |
The current SQLCLI SQLAppendBatch path processes a server batch as a transaction when no separate
active transaction exists. Constraint-error regression tests for this path roll back the entire
failing batch. Do not extend this guarantee to every SDK logical batch, multiple flushes, or an
Appender's entire lifetime as one atomic operation.
For AUTO_INCREMENT and DECIMAL input, also check the dedicated rules in
[SQLCLI and ODBC](/dbms/development-tools-integration/cli-odbc/). The Append protocol arrival-time
field does not create a LOG-style automatic timestamp column in TRANSACTION.
Before adoption, mix one duplicate-key or NULL-error row with valid rows and verify success/failure
counts and stored results. Retransmission after network errors must also account for duplicates of
already committed data.
---
title: "8.5 Query and Analysis"
url: https://docs.machbase.com/dbms/rdb-table-usage/query-analysis/
language: en
kind: page
---
# 8.5 Query and Analysis
Even syntactically valid SQL returns no rows when predicates differ from stored state values.
Insufficient sort keys can also make equal-timestamp rows appear in a different order between
queries. Check filtering, sorting, and aggregation with samples covering multiple states and time
boundaries.
## Prepare Example Data
```sql
CREATE TRANSACTION TABLE ch8_query (
order_id LONG PRIMARY KEY,
customer VARCHAR(32),
item_id LONG,
amount DECIMAL(18,2),
status VARCHAR(16),
ordered DATETIME
);
CREATE TRANSACTION TABLE ch8_query_product (id LONG PRIMARY KEY, name VARCHAR(64));
INSERT INTO ch8_query_product VALUES (42, 'Pump');
INSERT INTO ch8_query_product VALUES (43, 'Valve');
INSERT INTO ch8_query VALUES (
1001, 'C-01', 42, 10.25, 'PENDING', TO_DATE('2026-01-01 10:00:00', 'YYYY-MM-DD HH24:MI:SS'));
INSERT INTO ch8_query VALUES (
1002, 'C-01', 43, 20.50, 'PENDING', TO_DATE('2026-01-01 10:00:00', 'YYYY-MM-DD HH24:MI:SS'));
INSERT INTO ch8_query VALUES (
1003, 'C-02', 42, 30.75, 'SHIPPED', TO_DATE('2026-01-02', 'YYYY-MM-DD'));
SELECT order_id, amount, status FROM ch8_query WHERE order_id = 1001;
```
The query returns 1001, 10.25, PENDING.
## Time Predicates and Sorting
```sql
SELECT order_id, amount FROM ch8_query
WHERE ordered >= TO_DATE('2026-01-01', 'YYYY-MM-DD')
AND ordered < TO_DATE('2026-01-02', 'YYYY-MM-DD')
AND status = 'PENDING'
ORDER BY ordered DESC, order_id DESC
LIMIT 1;
```
Only 1002 is selected. order_id fixes ordering even when ordered timestamps match. Do not expect a
desired order from LIMIT alone. Use inclusive-start/exclusive-end predicates for consecutive daily
aggregates to avoid boundary duplicates. LOG-specific DURATION and automatic _arrival_time do not
apply to TRANSACTION.
## Joining Reference Data
```sql
SELECT o.order_id, p.name, o.amount
FROM ch8_query o JOIN ch8_query_product p ON o.item_id = p.id
WHERE o.customer = 'C-01'
ORDER BY o.order_id;
```
Results are (1001, Pump, 10.25) and (1002, Valve, 20.50). This INNER JOIN excludes orders without
reference data. Duplicate keys on the other side multiply results, so also check key uniqueness. For
joins with other table types, see [JOIN Examples](../join-relational-query/).
## Aggregate Queries
```sql
SELECT status, COUNT(*) AS cnt, SUM(amount) AS total_amount
FROM ch8_query GROUP BY status ORDER BY status;
```
PENDING has 2 rows totaling 30.75; SHIPPED has 1 row totaling 30.75. An index does not automatically
accelerate every aggregate. Check ranges, group counts, and result volume; consider separate
summaries for repeated workloads.
## JSON Path Queries
```sql
CREATE TRANSACTION TABLE ch8_query_json (id INTEGER PRIMARY KEY, state JSON);
INSERT INTO ch8_query_json VALUES (1, '{"status":"ALARM","score":90}');
INSERT INTO ch8_query_json VALUES (2, '{"status":"NORMAL","score":10}');
INSERT INTO ch8_query_json VALUES (3, '{"score":20}');
SELECT id, state->'$.status' AS status FROM ch8_query_json
WHERE state->'$.status' = 'ALARM'
ORDER BY id;
```
Only row 1 is selected. Include separate samples for missing paths and differing predicate values.
Distinguish arrow-path string comparisons from numeric extraction function comparisons. For
frequently used paths, consider
[JSON Path Indexes](../index-performance/#index-strategy-rdb-json-path).
## Indexes and Execution Plans
```sql
CREATE INDEX ch8_query_status_time ON ch8_query(status, ordered);
EXPLAIN SELECT order_id FROM ch8_query
WHERE status = 'PENDING'
AND ordered >= TO_DATE('2026-01-01', 'YYYY-MM-DD');
```
Align predicates with leading composite-index columns, but verify actual usage with EXPLAIN.
Assuming Machbase delegates entire relational queries unchanged to internal SQLite can lead to
incorrect interpretations of index selection and function predicates.
```sql
DROP TABLE ch8_query_json;
DROP TABLE ch8_query_product;
DROP TABLE ch8_query;
```
If results differ, check source row counts, WHERE predicates, and sort keys in order before
analyzing complex aggregates or joins.
---
title: "8.6 Indexes and Performance"
url: https://docs.machbase.com/dbms/rdb-table-usage/index-performance/
language: en
kind: page
---
# 8.6 Indexes and Performance
Indexes can enforce uniqueness as well as speed queries. Distinguish indexes protecting business
keys from those reducing reads so performance cleanup does not accidentally remove required
constraints.
## Index Types
| Type | Purpose | Composite columns | NULL |
|---|---|---|---|
| PRIMARY KEY | Row identification; one per table | Unsupported | Not allowed |
| UNIQUE INDEX | Business-key uniqueness | Supported | NULL-containing keys are not duplicates of each other |
| Ordinary index | Predicate-query access path | Supported | No uniqueness check |
TRANSACTION indexes are displayed as BTREE. Use a column PRIMARY KEY or add CREATE PRIMARY KEY INDEX
afterward. Do not apply LOG LSM/KEYWORD index syntax directly to TRANSACTION.
## UNIQUE INDEX and NULL
```sql
CREATE TRANSACTION TABLE ch8_index_account (
id LONG PRIMARY KEY,
email VARCHAR(120),
tenant INTEGER NOT NULL,
login VARCHAR(64)
);
INSERT INTO ch8_index_account VALUES (1, 'a@example.com', 1, 'alpha');
INSERT INTO ch8_index_account VALUES (2, 'b@example.com', 1, 'beta');
INSERT INTO ch8_index_account VALUES (3, NULL, 2, NULL);
INSERT INTO ch8_index_account VALUES (4, NULL, 2, NULL);
CREATE UNIQUE INDEX ch8_index_email ON ch8_index_account(email);
CREATE UNIQUE INDEX ch8_index_login ON ch8_index_account(tenant, login);
SELECT id FROM ch8_index_account WHERE email IS NULL ORDER BY id;
SHOW INDEX ch8_index_email;
```
Rows 3 and 4 both exist, and the index is created. If a business key must have values, every key
column needs NOT NULL as well as UNIQUE. Use separate CREATE UNIQUE INDEX instead of UNIQUE inside
CREATE TABLE.
The following two optional statements each test a uniqueness violation.
```sql
-- Expected failure: duplicate email
INSERT INTO ch8_index_account VALUES (5, 'a@example.com', 1, 'gamma');
UPDATE ch8_index_account SET email = 'a@example.com' WHERE id = 2;
```
After failure, four rows and row 2's b@example.com should remain. Ordinary UNIQUE violations
currently report ERR-01418. Check indexes and input values together rather than inferring the
duplicated business key from the error text alone.
## Dropping a UNIQUE INDEX
```sql
DROP INDEX ch8_index_email;
INSERT INTO ch8_index_account VALUES (5, 'a@example.com', 1, 'gamma');
SELECT id, email FROM ch8_index_account WHERE email = 'a@example.com' ORDER BY id;
```
Rows 1 and 5 are now both stored. Recreating the index while duplicates remain fails.
```sql
-- Expected failure: existing data contains duplicates.
CREATE UNIQUE INDEX ch8_index_email ON ch8_index_account(email);
```
Remove row 5 added during the exercise, then recreate the index.
```sql
DELETE FROM ch8_index_account WHERE id = 5;
CREATE UNIQUE INDEX ch8_index_email ON ch8_index_account(email);
SELECT COUNT(*) AS remaining_rows FROM ch8_index_account;
```
The count is 4. In production, determine whether an index provides performance or uniqueness before
removing it.
## Ordinary and Composite Indexes
```sql
CREATE TRANSACTION TABLE ch8_index_event (
id LONG PRIMARY KEY,
status VARCHAR(16),
created DATETIME,
state JSON
);
INSERT INTO ch8_index_event VALUES (
1, 'OPEN', TO_DATE('2026-01-01', 'YYYY-MM-DD'), '{"status":"ALARM","code":500}');
INSERT INTO ch8_index_event VALUES (
2, 'CLOSED', TO_DATE('2026-01-02', 'YYYY-MM-DD'), '{"status":"NORMAL","code":200}');
INSERT INTO ch8_index_event VALUES (
3, 'OPEN', TO_DATE('2026-01-03', 'YYYY-MM-DD'), '{"code":500}');
EXPLAIN SELECT id FROM ch8_index_event
WHERE status = 'OPEN' AND created >= TO_DATE('2026-01-01', 'YYYY-MM-DD');
CREATE INDEX ch8_index_status_time ON ch8_index_event(status, created);
EXPLAIN SELECT id FROM ch8_index_event
WHERE status = 'OPEN' AND created >= TO_DATE('2026-01-01', 'YYYY-MM-DD');
SELECT id FROM ch8_index_event
WHERE status = 'OPEN' AND created >= TO_DATE('2026-01-01', 'YYYY-MM-DD')
ORDER BY id;
```
Results must have the same semantics before and after index creation; the final query returns rows 1
and 3. Align the leading column with predicates, but do not assume every compound predicate uses the
index. Verify Machbase plan selection and supported predicate shapes with EXPLAIN. Timings for three
rows are not a benchmark.
## JSON Path Indexes
```sql
CREATE INDEX ch8_index_json_status ON ch8_index_event(state->'$.status');
CREATE INDEX ch8_index_json_code ON ch8_index_event(state->'$.code');
EXPLAIN SELECT id FROM ch8_index_event WHERE state->'$.status' = 'ALARM';
SELECT id FROM ch8_index_event WHERE state->'$.status' = 'ALARM' ORDER BY id;
SELECT id FROM ch8_index_event WHERE state->'$.code' = '500' ORDER BY id;
```
The status query returns row 1; the code query returns 1 and 3. An index does not mean every JSON
function expression uses that path. Distinguish result types and comparison semantics of arrow paths
and numeric extraction functions. For frequently repeated complex predicates, also compare a design
using ordinary extracted columns.
Even when a JSON path UNIQUE INDEX can be created, it is not a conflict-selection key for
TRANSACTION UPSERT. Check [UPSERT Constraints](../insert-on-duplicate-key-update/).
## Read and Write Costs
Compare with the same data volume, predicate values, and concurrent input rate before and after
index creation. Measure INSERT/UPDATE/DELETE throughput and index space as well as query time. LIMIT
reduces result volume, but ORDER BY is required to fix which rows are returned.
```sql
DROP TABLE ch8_index_event;
DROP TABLE ch8_index_account;
```
Replacing a UNIQUE INDEX with an ordinary index for performance removes uniqueness guarantees. First
verify that results and constraints remain equivalent across tuning changes.
---
title: "8.7 Operations and Data Lifecycle"
url: https://docs.machbase.com/dbms/rdb-table-usage/operations-lifecycle/
language: en
kind: page
---
# 8.7 Operations and Data Lifecycle
Deleting old business data requires more than checking dates. Cancelled orders and orders still in
progress have different retention criteria, and cleanup can conflict with other writes. Define
business predicates and processing units before deleting.
## Data Retention Criteria
Use TAG/LOG for source time series and TRANSACTION for mutable state or summary results. Their
retention periods need not match. TAG/LOG Retention Policy does not directly apply to TRANSACTION.
Design DELETE predicates and external scheduling for business requirements.
## Deletion and Rollback
```sql
CREATE TRANSACTION TABLE ch8_cleanup (
id LONG PRIMARY KEY,
status VARCHAR(16),
created DATETIME
);
INSERT INTO ch8_cleanup VALUES (1, 'CANCELLED', TO_DATE('2025-12-01', 'YYYY-MM-DD'));
INSERT INTO ch8_cleanup VALUES (2, 'PENDING', TO_DATE('2025-12-01', 'YYYY-MM-DD'));
INSERT INTO ch8_cleanup VALUES (3, 'CANCELLED', TO_DATE('2026-02-01', 'YYYY-MM-DD'));
BEGIN;
SELECT COUNT(*) AS delete_candidates FROM ch8_cleanup
WHERE status = 'CANCELLED' AND created < TO_DATE('2026-01-01', 'YYYY-MM-DD');
DELETE FROM ch8_cleanup
WHERE status = 'CANCELLED' AND created < TO_DATE('2026-01-01', 'YYYY-MM-DD');
SELECT id, status FROM ch8_cleanup ORDER BY id;
ROLLBACK;
SELECT COUNT(*) AS after_rollback FROM ch8_cleanup;
```
One row is targeted. Rows 2 and 3 remain during deletion; after rollback, the count is 3. The
exercise rolls back for verification. To finalize production work, choose COMMIT after business
approval and result checks. Close result cursors before ending the transaction.
Do not equate preliminary query counts with actual affected rows. Account for concurrent changes and
snapshot conflicts, and record affected counts returned during execution and final state. See
[Locks and Retries](../locking-conflict-timeout/).
## Batch Processing and Resumption
Partition deletion by date interval or unique-key range instead of one long transaction. Record
boundaries, processed counts, and commit results for each range so work can resume after an
interruption.
Also take care when copying reference data to another table before deleting the source. Do not
assume atomic cross-table commit under failure for multiple TRANSACTION tables. Design copy
verification and resumption according to [Transaction Guarantees](../transaction/). Avoid waiting
for external APIs or long file operations inside BEGIN.
## Backup Validation
Check business keys, row counts, totals, and indexes in the mounted backup, not merely
backup-command success. Use three-part names `mount_name.owner.table_name`. Two-part names can be
confused with production data.
Even incremental backups include a complete TRANSACTION table storage snapshot at the backup point.
Do not estimate additional space only from the changed row count. For exercises, see
[Backup, Restore, and Mount](../backup-restore-mount/).
## Checks After Cleanup
Fewer rows do not necessarily reduce operating-system file size immediately or proportionally. Check
business row counts, actual file usage, and retained backup volume separately. Do not connect
directly to internal SQLite files or delete files arbitrarily to reclaim space.
```sql
DROP TABLE ch8_cleanup;
```
If cleanup fails, check the last successful range and commit result before attempting further
deletion. These records reduce omissions and duplicate processing.
---
title: "8.8 Constraints, Errors, and Troubleshooting"
url: https://docs.machbase.com/dbms/rdb-table-usage/constraints-errors-troubleshooting/
language: en
kind: page
---
# 8.8 Constraints, Errors, and Troubleshooting
When migrating SQL from another RDBMS, similar names can suggest equivalent features incorrectly.
First check Edition and public syntax, then examine data constraints separately from concurrency
issues.
## Feature Support
TRANSACTION is Standard Edition only. Cluster rejects CREATE TABLE, CREATE TRANSACTION TABLE, and
CREATE TXN TABLE. Specify CREATE LOG TABLE explicitly for LOG.
| Requirement | Support or alternative |
|---|---|
| General SELECT/INSERT/UPDATE/DELETE | Supported; changes without WHERE target all rows |
| Single-column PRIMARY KEY | Supported; one per table |
| Single/composite UNIQUE INDEX | Supported; create separately after the table |
| Column UNIQUE or table-level PRIMARY KEY | These creation forms are unsupported |
| FOREIGN KEY, Trigger, Stored Procedure | Unsupported |
| BEGIN/COMMIT/ROLLBACK | Supported; nested BEGIN and SAVEPOINT unsupported |
| ADD/DROP/RENAME COLUMN, RENAME TO | Check supported conditions |
| MODIFY COLUMN | Unsupported |
| Append | Check SDK public paths and batch boundaries |
| TAG METADATA/BASETIME/BASEDISTANCE | Not applicable to TRANSACTION |
LOOKUP can support small reference-data changes in Cluster, but is not a complete replacement for
explicit relational transactions. Separate requirements appropriately, such as LOG/TAG for source
events and another RDBMS for relational transactions.
## Diagnosis by Error
| Symptom | What to check | Next action |
|---|---|---|
| ERR-01418 uniqueness violation | Primary/unique keys and existing data | Correct input or review UPSERT rules |
| NOT NULL violation | Omitted values, NULL, empty strings, and DEFAULT | Check input and constraints |
| UPDATE affects 0 rows | Key and current-state predicates | Distinguish missing/already-processed targets |
| Resource busy | Other writers, stale read snapshots, and open cursors | Wait, restart the transaction, or close cursors as appropriate |
| COMMIT/ROLLBACK is busy | Open result sets on the same connection | Close result sets and retry termination |
| Subsequent SQL is rejected after an error | Rollback-only state | ROLLBACK and start a new operation |
| DDL fails | Referencing indexes/views and active transactions | Adjust dependencies and timing |
| Mounted values differ from expectations | Mount, owner, and table names | Distinguish production from backup data |
Resource busy errors require different retry strategies. See the two-connection exercises in
[Locks and Busy Timeout](../locking-conflict-timeout/). Do not retry solely because the error text
contains TRANSACTION.
## Constraint Errors and Data Preservation
```sql
CREATE TRANSACTION TABLE ch8_error (
id LONG PRIMARY KEY,
code VARCHAR(32) NOT NULL,
value INTEGER
);
CREATE UNIQUE INDEX ch8_error_code ON ch8_error(code);
INSERT INTO ch8_error VALUES (1, 'A', 10);
```
Each optional example below intentionally fails. Run only the statement being checked, separately
from normal SQL.
```sql
-- Duplicate PRIMARY KEY
INSERT INTO ch8_error VALUES (1, 'B', 20);
-- Duplicate UNIQUE key
INSERT INTO ch8_error VALUES (2, 'A', 20);
-- Required-value violation
INSERT INTO ch8_error VALUES (3, NULL, 30);
-- Unsupported schema change
ALTER TABLE ch8_error MODIFY COLUMN (code VARCHAR(64));
```
```sql
SELECT id, code, value FROM ch8_error ORDER BY id;
DROP TABLE ch8_error;
```
The final query contains only (1, A, 10). Distinguish a failed statement preserving state from
automatic cancellation of earlier successful statements inside BEGIN; the latter does not follow.
See the comparison in [Transactions](../transaction/).
## Collecting Diagnostic Information
Collect server version/Edition, DDL and indexes, executed SQL, error code and full message, and
actual affected row counts. For connection failures, also record COMMIT request/response times and
business keys. Redact passwords, personal information, and sensitive business values; share only a
minimal reproducer.
Do not repair by editing internal storage files or recreating production tables. Establish the cause
and whether changes were applied first to reduce data loss.
---
title: "8.9 Transactions"
url: https://docs.machbase.com/dbms/rdb-table-usage/transaction/
language: en
kind: page
---
# 8.9 Transactions
When a statement fails midway through several SQL operations, earlier changes do not necessarily
disappear. Ordinary constraint errors distinguish the failing statement from the entire transaction.
This section first checks COMMIT/ROLLBACK boundaries using changes within one table.
## Executing Transactions
```sql
CREATE TRANSACTION TABLE ch8_tx (
item_id LONG PRIMARY KEY,
qty INTEGER NOT NULL
);
INSERT INTO ch8_tx VALUES (1, 10);
INSERT INTO ch8_tx VALUES (2, 20);
BEGIN;
UPDATE ch8_tx SET qty = qty - 3 WHERE item_id = 1 AND qty >= 3;
UPDATE ch8_tx SET qty = qty + 3 WHERE item_id = 2;
SELECT item_id, qty FROM ch8_tx ORDER BY item_id;
ROLLBACK;
SELECT item_id, qty FROM ch8_tx ORDER BY item_id;
```
Inside the transaction, values are 7 and 23; after ROLLBACK, they are 10 and 20. The application
must verify that each UPDATE affected the expected one row. Updating 0 rows because a predicate does
not match is not an SQL error, so the database does not automatically identify business failure.
The following successful exercise commits the same changes.
```sql
BEGIN;
UPDATE ch8_tx SET qty = qty - 3 WHERE item_id = 1 AND qty >= 3;
UPDATE ch8_tx SET qty = qty + 3 WHERE item_id = 2;
COMMIT;
SELECT item_id, qty FROM ch8_tx ORDER BY item_id;
```
Committed values are 7 and 23. The public syntax is BEGIN; BEGIN TRANSACTION, nested BEGIN, and
SAVEPOINT are unsupported. Without an explicit transaction, TRANSACTION DML is processed statement
by statement. Check driver autocommit and transaction APIs separately.
## Statement Errors and Rollback
This exercise demonstrates error handling. The duplicate INSERT intentionally fails. If the SQL tool
stops on error, ensure ROLLBACK is executed on the same connection.
```sql
BEGIN;
UPDATE ch8_tx SET qty = 100 WHERE item_id = 1;
```
```sql
-- Expected failure: duplicate item_id
INSERT INTO ch8_tx VALUES (1, 999);
```
```sql
SELECT item_id, qty FROM ch8_tx ORDER BY item_id;
ROLLBACK;
SELECT item_id, qty FROM ch8_tx ORDER BY item_id;
```
The first query shows 100 and 23; after ROLLBACK, 7 and 23. An ordinary constraint error rolls back
the failing statement, but earlier successful statements remain in the transaction. The application
must choose ROLLBACK to cancel the whole business operation. Continuing is not possible after every
error: a rollback-only state entered during recovery must be ended with ROLLBACK.
## TRUNCATE and Rollback
```sql
BEGIN;
TRUNCATE TABLE ch8_tx;
SELECT COUNT(*) AS during_truncate FROM ch8_tx;
ROLLBACK;
SELECT COUNT(*) AS after_rollback FROM ch8_tx;
```
Results are 0 and 2. Current TRANSACTION TRUNCATE deletes all rows as part of an explicit
transaction. Distinguish this from LOG/TAG cleanup and schema changes such as CREATE/ALTER/DROP.
Perform schema operations outside business transactions.
## Transaction Scope by Table Type
Queries and mixed joins involving LOG, TAG, LOOKUP, and VOLATILE are allowed during an active
TRANSACTION transaction. Writes to those types cannot be grouped into the same transaction.
Permitted reads do not give those types the same snapshot/rollback guarantees as TRANSACTION. Design
consistency between source ingestion and business-state changes separately.
TRANSACTION reads use snapshots that exclude other sessions' uncommitted changes. Do not assume
BEGIN fixes one common read point across every table at once. Read-to-write conflicts are covered in
[Locks and Retries](../locking-conflict-timeout/).
## Multitable Commit and Failures
DML on multiple TRANSACTION tables can be grouped with BEGIN and normally committed or rolled back.
However, storage currently uses per-table handles, and COMMIT processes those handles sequentially.
Do not interpret this as guaranteed atomic commit across all tables if a failure occurs during
commit.
Review this limitation first for workloads requiring indivisible multitable processing. After commit
failure or a lost response, sending ROLLBACK does not prove every table was restored. Check applied
state by business key. This is also why the basic exercise uses two rows in one table.
## Cursors and Transaction Termination
Open TRANSACTION cursors can cause COMMIT/ROLLBACK to fail with Resource busy. Close SDK result
sets/statements, then retry the termination command. Disconnecting rolls back uncommitted changes,
but the client must separately establish whether the server had already committed before the
connection was lost.
```sql
DROP TABLE ch8_tx;
```
Do not wait for external APIs or long computations inside BEGIN. Keep transactions short and
distinguish retrying an operation from first checking its result to make operational decisions
clearer.
---
title: "8.10 Locks, Conflicts, and Busy Timeout"
url: https://docs.machbase.com/dbms/rdb-table-usage/locking-conflict-timeout/
language: en
kind: page
---
# 8.10 Locks, Conflicts, and Busy Timeout
If different rows still produce Resource busy, thinking only in terms of row locks can obscure the
cause. TRANSACTION write conflicts can occur between different rows of the same table. Distinguish
conflicts resolved by waiting from those requiring a new transaction.
## Example Environment
This exercise targets a Standard validation environment with default WAL settings. A and B are
separate connections to the same account and database. Execute blocks in the stated order and fully
consume SELECT results to close cursors. Do not change production journal mode for this exercise.
Prepare the table in A.
```sql
SELECT NAME, VALUE FROM V$PROPERTY WHERE NAME = 'TRANSACTION_JOURNAL_MODE';
CREATE TRANSACTION TABLE ch8_lock (id INTEGER PRIMARY KEY, val INTEGER);
INSERT INTO ch8_lock VALUES (1, 10);
INSERT INTO ch8_lock VALUES (2, 20);
```
TRANSACTION_JOURNAL_MODE=4 means WAL. With another value, check the environment before expecting the
WAL snapshot results below.
## Concurrent Write Conflicts
In A, begin a transaction and leave it open.
```sql
BEGIN;
UPDATE ch8_lock SET val = 11 WHERE id = 1;
```
Next query in B. Set the wait time to 0 only on the example B connection.
```sql
ALTER SESSION SET TRANSACTION_BUSY_TIMEOUT_MS = 0;
SELECT id, val FROM ch8_lock ORDER BY id;
```
B sees the previous values 10 and 20, not A's uncommitted 11. B's following UPDATE intentionally
produces Resource busy.
```sql
-- B: failure is expected; a different row still writes the same table.
UPDATE ch8_lock SET val = val + 1 WHERE id = 2;
```
COMMIT in A, then repeat the UPDATE in B.
```sql
-- A
COMMIT;
```
```sql
-- B
UPDATE ch8_lock SET val = val + 1 WHERE id = 2;
SELECT id, val FROM ch8_lock ORDER BY id;
```
Values are now 11 and 21. This demonstrates why same-table write conflicts must not be interpreted
as conventional row-level locking.
## WAL Snapshot Conflicts
After completing the preceding steps, open a read transaction in A.
```sql
-- A
BEGIN;
SELECT val FROM ch8_lock WHERE id = 1;
```
After A reads 11, change the value in B. B has no explicit transaction active.
```sql
-- B
UPDATE ch8_lock SET val = val + 10 WHERE id = 1;
```
Switching A to a write now is expected to cause a snapshot conflict.
```sql
-- A: intentionally failing step
UPDATE ch8_lock SET val = val + 1 WHERE id = 1;
```
Another connection has already committed, so A cannot promote its stale read snapshot to a write.
Increasing busy timeout or setting it to -1 does not resolve this conflict by waiting. End A's
transaction and reassess the new state instead of repeating only UPDATE.
```sql
-- A
ROLLBACK;
BEGIN;
UPDATE ch8_lock SET val = val + 1 WHERE id = 1;
COMMIT;
SELECT id, val FROM ch8_lock ORDER BY id;
```
Final values are 22 and 21. If the next change was calculated from a value read earlier, repeat the
read and business decision in the new transaction.
## busy timeout
The server default TRANSACTION_BUSY_TIMEOUT_MS is 30000 ms and is copied to new sessions. Change the
current session with ALTER SESSION.
| Value | Handling of temporary lock conflicts |
|---|---|
| -1 | Wait until cancellation, disconnection, or lock release |
| 0 | Return busy without waiting |
| Positive | Wait up to that many milliseconds, then proceed or return busy |
Conflicts that retries cannot resolve, such as snapshot promotion conflicts, are exceptions to this
policy. -1 is not an infinite retry policy for every conflict. DDL_LOCK_TIMEOUT separately controls
DDL lock waits; changing it does not fix snapshot conflicts.
## Errors and Retries
Retrying because the message contains TRANSACTION also repeats type, constraint, and permission
errors. Use driver error codes, complete diagnostics, and operation type to identify retryable lock
conflicts.
To retry an explicit transaction, close open result sets, ROLLBACK, and rerun in a new transaction
within bounded attempts and total request time. Connection loss and lost COMMIT responses are
separate cases. Repeating counter increments or order processing without checking business keys for
prior completion can apply changes twice.
## Conflict Diagnosis
```sql
SELECT id, user_name, user_ip, transaction_busy_timeout_ms
FROM V$SESSION WHERE closed = 0 ORDER BY id;
SELECT id, sess_id, state, query FROM V$STMT
WHERE state LIKE 'Execute in progress%'
OR state LIKE 'Fetch in progress%';
```
This query does not directly map lock owners. V$MUTEX contains internal server mutex statistics, not
a business-row lock list. Check connection information together with application BEGIN/end records.
Do not make forced session termination the first action, because it can cancel uncommitted work.
Confirm that neither connection has an open transaction, then clean up in A.
```sql
DROP TABLE ch8_lock;
```
Closing example connections A and B removes the effect of B's session timeout. In production, moving
external API calls and long computations outside BEGIN can itself reduce waits.
---
title: "8.11 JOIN and Relational Query Design"
url: https://docs.machbase.com/dbms/rdb-table-usage/join-relational-query/
language: en
kind: page
---
# 8.11 JOIN and Relational Query Design
If adding JOIN changes order counts, first check relationship cardinality. INNER JOIN excludes
unmatched rows, while multiple matches multiply results. Error-free SQL does not guarantee correct
business counts.
## TRANSACTION–LOOKUP Joins
```sql
CREATE TRANSACTION TABLE ch8_join_order (
order_id LONG PRIMARY KEY,
item_id LONG,
qty INTEGER
);
CREATE LOOKUP TABLE ch8_join_product (id LONG PRIMARY KEY, name VARCHAR(64));
CREATE TRANSACTION TABLE ch8_join_payment (order_id LONG PRIMARY KEY, status VARCHAR(16));
INSERT INTO ch8_join_order VALUES (1, 42, 2);
INSERT INTO ch8_join_order VALUES (2, 99, 1);
INSERT INTO ch8_join_product VALUES (42, 'Pump');
INSERT INTO ch8_join_payment VALUES (1, 'PAID');
SELECT o.order_id, p.name, o.qty
FROM ch8_join_order o JOIN ch8_join_product p ON o.item_id = p.id
ORDER BY o.order_id;
SELECT o.order_id, p.name, o.qty
FROM ch8_join_order o LEFT JOIN ch8_join_product p ON o.item_id = p.id
ORDER BY o.order_id;
```
INNER JOIN returns only order 1; LEFT JOIN returns orders 1 and 2. Order 2's product name is NULL. A
foreign key does not block inserting an order for missing product 99, so design required reference
validation separately.
A common mistake is putting a right-side predicate in WHERE after LEFT JOIN. Adding
`WHERE p.name = 'Pump'`, for example, excludes NULL rows. Distinguish predicates for finding matches
from predicates filtering final results.
## Joins Between TRANSACTION Tables
```sql
SELECT o.order_id, o.qty, p.status
FROM ch8_join_order o
JOIN ch8_join_payment p ON o.order_id = p.order_id
ORDER BY o.order_id;
```
The query returns one row, (1, 2, PAID). Multiple payment-history rows per order would produce
multiple results. Check relationships and aggregation granularity to avoid duplicate totals when
summing order amounts after a join.
## TAG Joins by Nearby Time
This example joins every measurement within 5 seconds before or after an alarm.
```sql
CREATE TRANSACTION TABLE ch8_join_alarm (
alarm_id LONG PRIMARY KEY,
sensor VARCHAR(32),
occurred DATETIME
);
CREATE TAG TABLE ch8_join_sensor (
name VARCHAR(32) PRIMARY KEY,
time DATETIME BASETIME,
value DOUBLE SUMMARIZED
);
INSERT INTO ch8_join_alarm VALUES (
1, 'TEMP-01', TO_DATE('2026-01-01 10:00:05', 'YYYY-MM-DD HH24:MI:SS'));
INSERT INTO ch8_join_sensor VALUES (
'TEMP-01', TO_DATE('2026-01-01 10:00:00', 'YYYY-MM-DD HH24:MI:SS'), 10);
INSERT INTO ch8_join_sensor VALUES (
'TEMP-01', TO_DATE('2026-01-01 10:00:10', 'YYYY-MM-DD HH24:MI:SS'), 20);
INSERT INTO ch8_join_sensor VALUES (
'TEMP-01', TO_DATE('2026-01-01 10:00:11', 'YYYY-MM-DD HH24:MI:SS'), 30);
SELECT a.alarm_id, s.time, s.value
FROM ch8_join_alarm a JOIN ch8_join_sensor s ON a.sensor = s.name
WHERE s.time >= a.occurred - 5s
AND s.time <= a.occurred + 5s
ORDER BY a.alarm_id, s.time;
```
Alarm 1 joins to two rows with values 10 and 20. Both endpoints are included; value 30 is excluded.
This query does not select only the nearest measurement or an exact timestamp match. If one value is
required, separately define criteria such as latest preceding value or shortest distance, plus
tie-breaking rules.
## Join Design Criteria
Match join-key types and formats, restrict time ranges, then inspect the execution plan. Return only
required columns. Compare access paths before adding functions or conversions to join keys. Do not
assume join order or algorithms match another RDBMS.
Permitted mixed joins do not mean other table types share the same transaction snapshot as
TRANSACTION. Joining current LOOKUP descriptions also does not reproduce historical descriptions.
```sql
DROP TABLE ch8_join_sensor;
DROP TABLE ch8_join_alarm;
DROP TABLE ch8_join_payment;
DROP TABLE ch8_join_product;
DROP TABLE ch8_join_order;
```
If result counts differ, first compare counts before the join and matching-row counts per key. These
two checks clarify how the query needs to change.
---
title: "8.12 TRANSACTION Backup, Restore, and Mount"
url: https://docs.machbase.com/dbms/rdb-table-usage/backup-restore-mount/
language: en
kind: page
---
# 8.12 TRANSACTION Backup, Restore, and Mount
A successful backup command does not complete recovery preparation. When production and backup
tables have the same name, querying the wrong one can look like successful validation. This exercise
changes production values after backup and verifies that the two results differ.
## Backup and Restore Support
| Operation | TRANSACTION behavior |
|---|---|
| BACKUP DATABASE | Includes persistent TRANSACTION data in the target scope |
| BACKUP TABLE | Backs up the specified table and required metadata |
| Incremental backup | Includes a complete TRANSACTION storage snapshot at that backup point |
| MOUNT DATABASE | Queries the backup read-only |
| Offline instance restore | Stop the server and use machadmin -r |
| Online logical database restore | Restore a supported logical backup with RESTORE DATABASE |
Do not treat TRANSACTION data in incremental backups as a changed-row-only delta. Use supported
backup commands instead of copying internal files. A backup containing TRANSACTION is also not a
workaround for using those tables in Cluster.
## Backup and Mount Validation
This exercise uses SYS in a Standard validation environment. Backup/mount privileges and server file
permissions are required. Paths are server-side examples; use a new, nonexistent path for every run.
Check parent directories and free space. Do not delete existing backups merely to reuse paths.
```sql
CREATE TRANSACTION TABLE ch8_backup (
id LONG PRIMARY KEY,
code VARCHAR(32) NOT NULL,
amount DECIMAL(18,2)
);
CREATE UNIQUE INDEX ch8_backup_code ON ch8_backup(code);
INSERT INTO ch8_backup VALUES (1, 'A', 10.25);
INSERT INTO ch8_backup VALUES (2, 'B', 20.50);
BACKUP TABLE ch8_backup INTO DISK = '/backup/ch8_table_20260907_a';
UPDATE ch8_backup SET amount = 99.00 WHERE id = 1;
MOUNT DATABASE '/backup/ch8_table_20260907_a' TO ch8_bak;
SELECT id, code, amount FROM ch8_bak.SYS.ch8_backup ORDER BY id;
SELECT id, code, amount FROM ch8_backup ORDER BY id;
```
The backup contains 10.25 and 20.50; production contains 99.00 and 20.50. Mounted queries use
three-part names `mount_name.owner.table_name`. If another account owns the example, replace SYS
with the actual owner.
Running only `SELECT ... FROM ch8_backup` is a common mistake: it queries the current connection's
production table, not the mounted backup. Also do not expect tables excluded from the backup to be
present.
A mount is read-only, not a recovery environment for UPDATE or DDL. Finish verification, close open
cursors, and unmount.
```sql
UMOUNT DATABASE ch8_bak;
DROP TABLE ch8_backup;
```
This cleanup removes only the example table and mount. The backup directory remains; manage it
separately under the retention policy.
## Restore Validation Checks
In an isolated restore environment, verify owners, row counts, business keys, monetary totals, and
representative JSON values. Check retained PRIMARY KEY/UNIQUE INDEX definitions, required
privileges, and application COMMIT/ROLLBACK flows. Mounted read-only validation does not establish
that restored writes work.
Online RESTORE DATABASE follows logical-backup and target-database requirements. Distinguish it from
a complete instance image containing multiple databases. Offline restore replacing an existing
instance and REPLACE are outside this exercise. Check privileges, downtime, and target-replacement
requirements in [Restore Syntax](/dbms/reference/sql/syntax/backup-restore-mount-syntax/) and
[Operational Procedures](/dbms/operations-configuration-recovery/backup-restore-mount/) before
executing them separately.
For business consistency across tables, do not rely on backup-command success alone. Define
write-pausing procedures, the business reference point, and cross-table validation criteria
together.
---
title: "8.13 TRANSACTION INSERT ON DUPLICATE KEY UPDATE"
url: https://docs.machbase.com/dbms/rdb-table-usage/insert-on-duplicate-key-update/
language: en
kind: page
---
# 8.13 TRANSACTION INSERT ON DUPLICATE KEY UPDATE
Inserting when absent and updating when present seems simple, but the definition of a duplicate
determines which row changes. Retransmitting counter increments can apply one event twice. Check
keys, update targets, and retry policy together.
TRANSACTION `INSERT ... ON DUPLICATE KEY UPDATE` updates an existing row on a PRIMARY KEY or
ordinary UNIQUE INDEX conflict. Exercises create their own objects. Keep error-checking SQL separate
from normal flow and complete cleanup before rerunning.
## Supported Syntax
TRANSACTION supports the `INSERT ... VALUES ...` UPSERT form.
```text
INSERT INTO table_name VALUES (...)
ON DUPLICATE KEY UPDATE;
INSERT INTO table_name VALUES (...)
ON DUPLICATE KEY UPDATE SET column_name = expression [, ...];
INSERT INTO table_name(column_name, ...)
VALUES (...)
ON DUPLICATE KEY UPDATE;
INSERT INTO table_name(column_name, ...)
VALUES (...)
ON DUPLICATE KEY UPDATE SET column_name = expression [, ...];
```
The following keys qualify for conflict detection.
- TRANSACTION PRIMARY KEY
- TRANSACTION UNIQUE INDEX
- TRANSACTION composite UNIQUE INDEX
## Basic Behavior
Without a duplicate, a new row is added as in ordinary INSERT.
```sql
CREATE TRANSACTION TABLE ch8_up_device_state (
device_id INTEGER PRIMARY KEY,
status VARCHAR(16),
alarm_count INTEGER,
updated_at DATETIME
);
INSERT INTO ch8_up_device_state
VALUES (1, 'NORMAL', 0, TO_DATE('2026-07-10 09:00:00', 'YYYY-MM-DD HH24:MI:SS'))
ON DUPLICATE KEY UPDATE SET
status = 'NORMAL',
updated_at = TO_DATE('2026-07-10 09:00:00', 'YYYY-MM-DD HH24:MI:SS');
```
A duplicate PRIMARY KEY updates the existing row.
```sql
INSERT INTO ch8_up_device_state
VALUES (1, 'ALARM', 1, TO_DATE('2026-07-10 09:05:00', 'YYYY-MM-DD HH24:MI:SS'))
ON DUPLICATE KEY UPDATE SET
status = 'ALARM',
alarm_count = alarm_count + 1,
updated_at = TO_DATE('2026-07-10 09:05:00', 'YYYY-MM-DD HH24:MI:SS');
```
The `SET` clause cannot change the PRIMARY KEY. Right-hand expressions are evaluated against the
conflicting existing row. Thus `alarm_count = alarm_count + 1` adds 1 to the existing alarm_count,
not the attempted insert value.
```sql
SELECT device_id, status, alarm_count, updated_at
FROM ch8_up_device_state
WHERE device_id = 1;
```
The expected result has the following form.
```text
DEVICE_ID STATUS ALARM_COUNT UPDATED_AT
--------- ------ ----------- -----------------------------
1 ALARM 1 2026-07-10 09:05:00 000:000:000
```
The `SET` clause after `ON DUPLICATE KEY UPDATE` is optional.
```sql
CREATE TRANSACTION TABLE ch8_up_asset_cache (
asset_id INTEGER PRIMARY KEY,
asset_name VARCHAR(80),
location VARCHAR(80),
keep_value INTEGER
);
INSERT INTO ch8_up_asset_cache VALUES (1, 'compressor-a', 'plant-1', 100);
INSERT INTO ch8_up_asset_cache(asset_id, asset_name, location)
VALUES (1, 'compressor-a-renamed', 'plant-2')
ON DUPLICATE KEY UPDATE;
```
Without SET, only non-PRIMARY KEY columns included in the INSERT target list are applied to the
existing row. Here, asset_name and location change; keep_value, omitted from the list, retains its
previous value.
```sql
SELECT asset_id, asset_name, location, keep_value
FROM ch8_up_asset_cache
ORDER BY asset_id;
```
The expected result has the following form.
```text
ASSET_ID ASSET_NAME LOCATION KEEP_VALUE
-------- -------------------- -------- ----------
1 compressor-a-renamed plant-2 100
```
For a table containing only a PRIMARY KEY column, a duplicate UPSERT without SET has no columns to
update, so the row remains unchanged.
## UNIQUE INDEX Duplicate Handling
UNIQUE INDEX conflicts also take the update path, not only PRIMARY KEY conflicts.
```sql
CREATE TRANSACTION TABLE ch8_up_account_profile (
id INTEGER PRIMARY KEY,
email VARCHAR(120),
display_name VARCHAR(80),
login_count INTEGER
);
CREATE UNIQUE INDEX ch8_up_uidx_account_profile_email
ON ch8_up_account_profile(email);
INSERT INTO ch8_up_account_profile
VALUES (1, 'ops@example.com', 'ops-user', 1);
INSERT INTO ch8_up_account_profile
VALUES (2, 'ops@example.com', 'ops-renamed', 1)
ON DUPLICATE KEY UPDATE SET
display_name = 'ops-renamed',
login_count = login_count + 1;
SELECT id, email, display_name, login_count
FROM ch8_up_account_profile
ORDER BY id;
```
The email UNIQUE INDEX conflicts, so the row with id=1 is updated. The attempted id=2 row is not inserted.
A composite UNIQUE INDEX treats the entire matching key combination as a duplicate.
```sql
CREATE TRANSACTION TABLE ch8_up_daily_device_summary (
id INTEGER PRIMARY KEY,
device_id INTEGER,
summary_day VARCHAR(10),
event_count INTEGER,
last_status VARCHAR(16)
);
CREATE UNIQUE INDEX ch8_up_uidx_daily_device_summary
ON ch8_up_daily_device_summary(device_id, summary_day);
INSERT INTO ch8_up_daily_device_summary
VALUES (1, 101, '2026-07-10', 3, 'NORMAL');
INSERT INTO ch8_up_daily_device_summary
VALUES (2, 101, '2026-07-10', 1, 'ALARM')
ON DUPLICATE KEY UPDATE SET
event_count = event_count + 1,
last_status = 'ALARM';
INSERT INTO ch8_up_daily_device_summary
VALUES (3, 101, '2026-07-11', 1, 'NORMAL')
ON DUPLICATE KEY UPDATE SET
event_count = event_count + 1;
```
The first UPSERT updates `(device_id, summary_day) = (101, '2026-07-10')`. The second inserts a new
row because the date differs.
UNIQUE keys containing NULL are not treated as duplicates of each other. These two inserts create
separate rows instead of overwriting each other.
```sql
INSERT INTO ch8_up_daily_device_summary VALUES (4, NULL, '2026-07-10', 1, 'NORMAL')
ON DUPLICATE KEY UPDATE;
INSERT INTO ch8_up_daily_device_summary VALUES (5, NULL, '2026-07-10', 2, 'ALARM')
ON DUPLICATE KEY UPDATE;
SELECT id, device_id, summary_day, event_count
FROM ch8_up_daily_device_summary ORDER BY id;
```
Result IDs are 1, 3, 4, and 5, with event_count values 4, 1, 1, and 2, respectively. Required
business keys need NOT NULL on their component columns as well as a UNIQUE INDEX.
## Usage Examples
TAG can continuously store time-ordered measurements while TRANSACTION maintains only the latest
state per device.
```sql
CREATE TRANSACTION TABLE ch8_up_latest_device_status (
device_name VARCHAR(80) PRIMARY KEY,
last_value DOUBLE,
last_state VARCHAR(16),
event_count LONG,
updated_at DATETIME
);
INSERT INTO ch8_up_latest_device_status
VALUES ('compressor-a', 72.5, 'NORMAL', 1, TO_DATE('2026-07-10 09:00:00', 'YYYY-MM-DD HH24:MI:SS'))
ON DUPLICATE KEY UPDATE SET
last_value = 72.5,
last_state = 'NORMAL',
event_count = event_count + 1,
updated_at = TO_DATE('2026-07-10 09:00:00', 'YYYY-MM-DD HH24:MI:SS');
INSERT INTO ch8_up_latest_device_status
VALUES ('compressor-a', 91.2, 'ALARM', 1, TO_DATE('2026-07-10 09:05:00', 'YYYY-MM-DD HH24:MI:SS'))
ON DUPLICATE KEY UPDATE SET
last_value = 91.2,
last_state = 'ALARM',
event_count = event_count + 1,
updated_at = TO_DATE('2026-07-10 09:05:00', 'YYYY-MM-DD HH24:MI:SS');
```
Use this pattern when dashboards need fast access only to current state.
When an external system repeatedly sends reference data with the same business key, UPSERT can use
its UNIQUE INDEX.
```sql
CREATE TRANSACTION TABLE ch8_up_customer_device (
id LONG PRIMARY KEY AUTO_INCREMENT,
external_device_id VARCHAR(64),
device_name VARCHAR(80),
owner_name VARCHAR(80),
enabled INTEGER
);
CREATE UNIQUE INDEX ch8_up_uidx_customer_device_external_id
ON ch8_up_customer_device(external_device_id);
INSERT INTO ch8_up_customer_device(external_device_id, device_name, owner_name, enabled)
VALUES ('ERP-DEV-10001', 'compressor-a', 'line-1', 1)
ON DUPLICATE KEY UPDATE SET
device_name = 'compressor-a',
owner_name = 'line-1',
enabled = 1;
INSERT INTO ch8_up_customer_device(external_device_id, device_name, owner_name, enabled)
VALUES ('ERP-DEV-10001', 'compressor-a-renamed', 'line-2', 1)
ON DUPLICATE KEY UPDATE SET
device_name = 'compressor-a-renamed',
owner_name = 'line-2',
enabled = 1;
```
The first INSERT creates a row; the second updates it through an external_device_id UNIQUE INDEX
conflict. The internal id remains unchanged.
Use UPSERT without SET to apply source column values directly to a current-state cache.
```sql
CREATE TRANSACTION TABLE ch8_up_tag_alias_cache (
alias_name VARCHAR(80) PRIMARY KEY,
tag_name VARCHAR(80),
unit VARCHAR(16),
description VARCHAR(160),
manually_checked INTEGER
);
INSERT INTO ch8_up_tag_alias_cache
VALUES ('compressor-a-temp', 'comp_a.temp', 'celsius', 'main compressor temp', 1);
INSERT INTO ch8_up_tag_alias_cache(alias_name, tag_name, unit, description)
VALUES ('compressor-a-temp', 'comp_a.temperature', 'celsius', 'renamed tag')
ON DUPLICATE KEY UPDATE;
```
This statement updates only tag_name, unit, and description. manually_checked is absent from the
column list and retains its value.
An aggregate table can accumulate occurrence counts by key.
```sql
CREATE TRANSACTION TABLE ch8_up_alarm_counter (
alarm_code VARCHAR(32) PRIMARY KEY,
first_seen DATETIME,
last_seen DATETIME,
hit_count LONG
);
INSERT INTO ch8_up_alarm_counter
VALUES (
'OVER_TEMP',
TO_DATE('2026-07-10 09:00:00', 'YYYY-MM-DD HH24:MI:SS'),
TO_DATE('2026-07-10 09:00:00', 'YYYY-MM-DD HH24:MI:SS'),
1
)
ON DUPLICATE KEY UPDATE SET
last_seen = TO_DATE('2026-07-10 09:00:00', 'YYYY-MM-DD HH24:MI:SS'),
hit_count = hit_count + 1;
INSERT INTO ch8_up_alarm_counter
VALUES (
'OVER_TEMP',
TO_DATE('2026-07-10 09:05:00', 'YYYY-MM-DD HH24:MI:SS'),
TO_DATE('2026-07-10 09:05:00', 'YYYY-MM-DD HH24:MI:SS'),
1
)
ON DUPLICATE KEY UPDATE SET
last_seen = TO_DATE('2026-07-10 09:05:00', 'YYYY-MM-DD HH24:MI:SS'),
hit_count = hit_count + 1;
```
`hit_count = hit_count + 1` is evaluated against the existing row, making it suitable for cumulative
counters.
JSON columns can also be update targets.
```sql
CREATE TRANSACTION TABLE ch8_up_device_json_state (
device_id INTEGER PRIMARY KEY,
state JSON,
updated_at DATETIME
);
INSERT INTO ch8_up_device_json_state
VALUES (
1,
'{"status":"NORMAL","score":10}',
TO_DATE('2026-07-10 09:00:00', 'YYYY-MM-DD HH24:MI:SS')
)
ON DUPLICATE KEY UPDATE SET
state = '{"status":"NORMAL","score":10}',
updated_at = TO_DATE('2026-07-10 09:00:00', 'YYYY-MM-DD HH24:MI:SS');
INSERT INTO ch8_up_device_json_state
VALUES (
1,
'{"status":"ALARM","score":90}',
TO_DATE('2026-07-10 09:05:00', 'YYYY-MM-DD HH24:MI:SS')
)
ON DUPLICATE KEY UPDATE SET
state = '{"status":"ALARM","score":90}',
updated_at = TO_DATE('2026-07-10 09:05:00', 'YYYY-MM-DD HH24:MI:SS');
SELECT JSON_EXTRACT_STRING(state, '$.status') AS status,
JSON_EXTRACT_INTEGER(state, '$.score') AS score
FROM ch8_up_device_json_state
WHERE device_id = 1;
```
Limitation: JSON path UNIQUE INDEX is excluded from conflict-key candidates. Its conflicts produce
uniqueness errors instead of taking the UPSERT update path.
## Transactions and Privileges
TRANSACTION UPSERT is committed or rolled back inside a transaction like ordinary INSERT/UPDATE.
```sql
CREATE TRANSACTION TABLE ch8_up_tx_device_state (
id INTEGER PRIMARY KEY,
status VARCHAR(16),
count_value INTEGER
);
INSERT INTO ch8_up_tx_device_state VALUES (1, 'NORMAL', 10);
BEGIN;
INSERT INTO ch8_up_tx_device_state VALUES (2, 'NORMAL', 1)
ON DUPLICATE KEY UPDATE SET count_value = count_value + 1;
INSERT INTO ch8_up_tx_device_state VALUES (1, 'ALARM', 1)
ON DUPLICATE KEY UPDATE SET
status = 'ALARM',
count_value = count_value + 1;
ROLLBACK;
SELECT id, status, count_value
FROM ch8_up_tx_device_state
ORDER BY id;
```
Both insert and update paths are rolled back in this example.
If a duplicate-update statement fails with an ordinary constraint violation, earlier successful
changes can remain in the explicit transaction. After checking the error, the application must
decide whether to continue or cancel business changes with ROLLBACK.
TRANSACTION UPSERT requires both INSERT and UPDATE privileges. Grant both even when execution takes
the insert path, because the statement includes an update path.
The following shows only the privilege syntax. Apply it as a separate administrative operation using
the actual owner, table, and existing application account.
```text
GRANT INSERT ON owner.table_name TO app_user;
GRANT UPDATE ON owner.table_name TO app_user;
```
SELECT privilege is not required to execute TRANSACTION UPSERT itself. It is required separately if
the application executes SELECT to verify results.
## Supported Types and Constraints
Columns updated in SET follow the same public type support as ordinary TRANSACTION UPDATE.
| Category | Types |
| --- | --- |
| Integer | `SHORT`, `INT16`, `USHORT`, `UINT16`, `INT`, `INTEGER`, `INT32`, `UINTEGER`, `UINT32`, `LONG`, `INT64`, `ULONG`, `UINT64` |
| Floating point | `FLOAT`, `DOUBLE` |
| Fixed point | `DECIMAL`, `NUMERIC`, `DEC`, `FIXED`, `NUMBER` |
| String/LOB | `VARCHAR`, `TEXT`, `CLOB`, `BINARY`, `BLOB` |
| Other | `DATETIME`, `IPV4`, `IPV6`, `JSON` |
This table lists scalar types used by TRANSACTION. For numeric ARRAY support, see the
[Data Type Dictionary](/dbms/reference/sql/types/).
Conflict keys and index types follow TRANSACTION PRIMARY KEY and UNIQUE INDEX type policies. UPSERT
does not expand supported key types.
The following syntax is unsupported.
```text
-- Illustrative examples of unsupported syntax.
-- Combining INSERT SELECT with ON DUPLICATE KEY UPDATE is unsupported.
INSERT INTO ch8_up_device_state(device_id, status, alarm_count, updated_at)
SELECT device_id, status, alarm_count, updated_at
FROM staging_device_state
ON DUPLICATE KEY UPDATE SET status = 'UPDATED';
-- The MySQL VALUES(col) function is unsupported.
INSERT INTO ch8_up_device_state VALUES (1, 'ALARM', 1, TO_DATE('2026-07-10 09:00:00', 'YYYY-MM-DD HH24:MI:SS'))
ON DUPLICATE KEY UPDATE SET status = VALUES(status);
-- The EXCLUDED alias is unsupported.
INSERT INTO ch8_up_device_state VALUES (1, 'ALARM', 1, TO_DATE('2026-07-10 09:00:00', 'YYYY-MM-DD HH24:MI:SS'))
ON DUPLICATE KEY UPDATE SET status = EXCLUDED.status;
-- Conflict-target syntax is unsupported.
INSERT INTO ch8_up_device_state VALUES (1, 'ALARM', 1, TO_DATE('2026-07-10 09:00:00', 'YYYY-MM-DD HH24:MI:SS'))
ON CONFLICT (device_id) DO UPDATE SET status = 'ALARM';
```
Target-table restrictions are as follows.
- This page covers only TRANSACTION PRIMARY KEY/UNIQUE conflict behavior. The same SQL form also supports PRIMARY KEY conflicts in LOOKUP and VOLATILE. Use the [DML Dictionary](/dbms/reference/sql/syntax/dml-syntax/#on-duplicate-key-update) as the authoritative common syntax reference.
- LOG and TAG DATA rows are unsupported. For TAG METADATA tag-name conflicts, see the [DML Dictionary](/dbms/reference/sql/syntax/dml-syntax/#on-duplicate-key-update).
- Even a TRANSACTION table requires a PRIMARY KEY or UNIQUE INDEX for UPSERT.
- JSON path UNIQUE INDEX is not used as a conflict key.
## Conflicts and Error Handling
If multiple UNIQUE indexes identify the same existing row, it is updated once. If they conflict with
different existing rows, the statement fails because it cannot determine which row to update.
The following sample conflicts with two business keys on different rows.
```sql
CREATE TRANSACTION TABLE ch8_up_user_contact (
id INTEGER PRIMARY KEY,
email VARCHAR(120),
phone VARCHAR(40),
note VARCHAR(80)
);
CREATE UNIQUE INDEX ch8_up_uidx_user_contact_email ON ch8_up_user_contact(email);
CREATE UNIQUE INDEX ch8_up_uidx_user_contact_phone ON ch8_up_user_contact(phone);
INSERT INTO ch8_up_user_contact VALUES (1, 'a@example.com', '010-0000-0001', 'user-a');
INSERT INTO ch8_up_user_contact VALUES (2, 'b@example.com', '010-0000-0002', 'user-b');
```
Only the following INSERT intentionally fails in this optional exercise.
```sql
-- email conflicts with id=1; phone conflicts with id=2.
-- Different conflicting rows cause failure instead of selecting an update path.
INSERT INTO ch8_up_user_contact VALUES (3, 'a@example.com', '010-0000-0002', 'ambiguous')
ON DUPLICATE KEY UPDATE SET note = 'updated';
```
If SET results violate another UNIQUE constraint or NOT NULL, the statement fails and preserves the
existing row.
## Retransmission and Current State
Counter-increment UPSERT is not automatic deduplication. Repeating the same event increments the
counter again. If a COMMIT response is lost, first verify the result using business keys and
event-processing records.
Simple overwrites maintain the latest state only when ingestion order matches event order. Define
source-time comparison and collection policies so late historical events do not overwrite newer
values. For changes spanning tables, also check [Commit Scope During Failures](../transaction/).
## Operational Recommendations
- Define PRIMARY KEY or UNIQUE INDEX first when the business key is clear.
- Use `SET count_col = count_col + 1` for cumulative counters.
- Use UPSERT without SET to apply source row values directly. Columns omitted from the list retain their values.
- With multiple UNIQUE indexes, handle inputs that could conflict with different existing rows before execution.
- When migrating MySQL-compatible SQL, replace VALUES(col), EXCLUDED, and ON CONFLICT with supported Machbase syntax.
- Avoid JSON path UNIQUE INDEX as an UPSERT key. If needed, extract the key into an ordinary column and create a UNIQUE INDEX there.
## Verify Results and Clean Up
After completing the successful exercise, recheck representative values and counts. The expectations
below assume every normal statement ran once, excluding intentional errors.
```sql
SELECT id, email, display_name, login_count
FROM ch8_up_account_profile ORDER BY id;
SELECT device_name, last_state, event_count
FROM ch8_up_latest_device_status;
SELECT external_device_id, device_name, owner_name
FROM ch8_up_customer_device;
SELECT alias_name, tag_name, manually_checked FROM ch8_up_tag_alias_cache;
SELECT alarm_code, hit_count FROM ch8_up_alarm_counter;
SELECT id, status, count_value FROM ch8_up_tx_device_state ORDER BY id;
SELECT id, note FROM ch8_up_user_contact ORDER BY id;
```
| Sample | Expected result |
|---|---|
| account_profile | id=1 retained, display_name=ops-renamed, login_count=2 |
| latest_device_status | ALARM, event_count=2 |
| customer_device | One external-key row; name compressor-a-renamed, owner line-2 |
| tag_alias_cache | tag_name=comp_a.temperature; manually_checked=1 retained |
| alarm_counter | OVER_TEMP hit_count=2 |
| tx_device_state | After ROLLBACK, only existing id=1, NORMAL, count_value=10 |
| user_contact | user-a and user-b at id=1 and id=2 remain unchanged |
```sql
DROP TABLE ch8_up_user_contact;
DROP TABLE ch8_up_tx_device_state;
DROP TABLE ch8_up_device_json_state;
DROP TABLE ch8_up_alarm_counter;
DROP TABLE ch8_up_tag_alias_cache;
DROP TABLE ch8_up_customer_device;
DROP TABLE ch8_up_latest_device_status;
DROP TABLE ch8_up_daily_device_summary;
DROP TABLE ch8_up_account_profile;
DROP TABLE ch8_up_asset_cache;
DROP TABLE ch8_up_device_state;
```
DROP targets only this page's example objects. For complex keys, compare attempted input with
existing conflicting rows side by side. Clarifying which row was intended for update helps identify
the cause.
---
title: "9. LOOKUP Table Usage"
url: https://docs.machbase.com/dbms/lookup-table-usage/
language: en
kind: section
---
# 9. LOOKUP Table Usage
LOOKUP tables load persistently stored reference and master data into memory at server startup,
providing fast access by PRIMARY KEY. This chapter covers the memory-resident architecture, JSON and
SEQUENCE, JOIN, and DML with general predicates.
## Chapter Contents
| Section | Topics |
|----|------|
| [Overview and Use Criteria](./overview-use-criteria/) | Purpose and selection criteria |
| [Table Structure and Schema](./table-structure-schema/) | Columns, primary keys, and schema design |
| [Create, Alter, and Drop](./create-alter-drop/) | CREATE, ALTER, and DROP DDL |
| [Data Ingestion and Modification](./data-input-mutation/) | INSERT, Append, reload, and deletion |
| [Queries and Analysis](./query-analysis/) | SELECT, JOIN, and predicate queries |
| [Indexes and Performance](./index-performance/) | Red-black indexes, secondary indexes, and tuning |
| [Operations and Data Lifecycle](./operations-lifecycle/) | Backup, recovery, and persistence |
| [Constraints, Errors, and Troubleshooting](./constraints-errors-troubleshooting/) | Limitations, error causes, and remedies |
| [Usage Patterns and Scenarios](./patterns-scenarios/) | Code tables, reference data, and thresholds |
| [PRIMARY KEY Policy](./primary-key-policy/) | Natural versus surrogate keys and key immutability |
| [SEQUENCE Columns](./sequence-column/) | Automatic numbering and NEXTVAL |
| [JSON Columns and Queries](./json-column-query/) | Supported JSON features, path predicates, and primary key restrictions |
| [UPDATE/DELETE with General Predicates](./predicate-update-delete/) | Modifications using non-PK, range, string, date, and JSON path predicates |
---
title: "9.1 Overview and Use Criteria"
url: https://docs.machbase.com/dbms/lookup-table-usage/overview-use-criteria/
language: en
kind: page
---
# 9.1 Overview and Use Criteria
LOOKUP tables store relatively small, frequently referenced datasets such as code lists, device
master data, thresholds, and settings. Data is persistent, but all rows used by SQL queries reside
in memory. This makes them suitable for repeated key-based lookups and updates.
## LOOKUP Table Characteristics
Create a LOOKUP table with `CREATE LOOKUP TABLE`. A `PRIMARY KEY` is required.
```sql
CREATE LOOKUP TABLE ch9_overview (
sensor_id VARCHAR(64) PRIMARY KEY,
site VARCHAR(32),
unit VARCHAR(16),
status VARCHAR(16)
);
```
LOOKUP tables have the following characteristics.
| Item | Description |
|------|------|
| Main uses | Code tables, device master data, thresholds, and reference data |
| Requirement | `PRIMARY KEY` required |
| Storage | Persistent storage; all rows loaded into memory at server startup |
| Query patterns | Optimized for primary key lookups; supports general predicates and JOIN with other tables |
| Modification patterns | INSERT, UPDATE, DELETE |
| Additional features | SEQUENCE columns and Append duplicate-key policy |
## Use Criteria
Use a LOOKUP table when:
- The dataset is relatively small and frequently referenced as a whole.
- All rows and required secondary indexes fit in server memory.
- You manage reference information such as codes, names, locations, units, or states.
- Descriptive information must be joined to source data in TAG or LOG tables.
- Reference values, such as thresholds or settings, can change during operation.
- A `PRIMARY KEY` clearly identifies each row.
For a JOIN that adds descriptions such as location and unit to TAG or LOG data, see
[Queries and Analysis](/dbms/lookup-table-usage/query-analysis/).
## When to Consider Other Tables
Consider other table types for the following requirements.
| Requirement | Recommended table |
|----------|-------------|
| Large volumes of time-series measurements | TAG |
| Append-oriented source events | LOG |
| Large relational datasets that cannot all fit in memory | TRANSACTION |
| Transactions and relational business processing | TRANSACTION |
| Current-state cache kept only in server memory | VOLATILE |
LOOKUP is suitable for reference data. Persistence does not make it a disk-oriented large-volume
table. Store source data in LOG or TAG, and use LOOKUP for memory-resident reference data queried
repeatedly. Use TRANSACTION when relational data exceeds available memory or requires complex
business processing.
Clean up the example table as follows.
```sql
DROP TABLE ch9_overview;
```
## Design Sequence
Make the following decisions when designing a LOOKUP table.
1. Choose the `PRIMARY KEY` that identifies each row.
2. Choose a natural key or a surrogate key based on SEQUENCE or AUTO_INCREMENT.
3. Add indexes to columns frequently queried or joined.
4. Validate memory requirements for expected row sizes, row counts, and secondary indexes.
5. Distinguish columns updated during operation from immutable columns.
6. Prepare a query that checks the target scope before bulk changes.
For schema and key design, see
[Table Structure and Schema](/dbms/lookup-table-usage/table-structure-schema/) and
[PRIMARY KEY Policy](/dbms/lookup-table-usage/primary-key-policy/).
---
title: "9.2 Table Structure and Schema"
url: https://docs.machbase.com/dbms/lookup-table-usage/table-structure-schema/
language: en
kind: page
---
# 9.2 Table Structure and Schema
This section covers LOOKUP table structure and schema design.
## LOOKUP Table Design
LOOKUP tables store code lists and reference data. A PRIMARY KEY identifies each row. They support
UPDATE/DELETE by primary key or general predicates and store data persistently on disk.
### Persistent Storage and In-Memory Queries
LOOKUP uses two layers to provide persistence and in-memory query performance.
1. Modified rows are written to persistent storage so they survive restarts.
2. At startup, the server reads all LOOKUP rows from persistent storage and reconstructs in-memory rows containing all column values.
3. Each in-memory row is registered in the mandatory PRIMARY KEY red-black index.
4. SQL queries use the reconstructed in-memory rows and indexes.
Conceptually, each row can be viewed as a key-value entry.
```
PRIMARY KEY Other column values
sensor_id = 'TEMP-01' ───────► { site, unit, status, ... }
key value
```
This explains why LOOKUP is particularly suitable for PRIMARY KEY lookups while also providing a SQL
table interface, general predicate queries, JOIN, and secondary indexes. Persistent storage does not
mean that only requested rows are fetched from disk during queries. All rows and red-black secondary
indexes consume memory. Consider variable-length columns, JSON values, and secondary index sizes as
well as row counts when designing the schema.
- **[Use Cases](/dbms/lookup-table-usage/patterns-scenarios/#use-cases-lookup)**
- **[PRIMARY KEY Design](/dbms/lookup-table-usage/primary-key-policy/#design-primary-key)**
- **[Column and Sequence Design](/dbms/lookup-table-usage/sequence-column/#design-column-lookup-sequence)**
- **[JSON Columns and Queries](/dbms/lookup-table-usage/json-column-query/#condition-query-lookup-json)**
- **[Reference Design Patterns](/dbms/lookup-table-usage/patterns-scenarios/#patterns-reference-design)**
- **[Index Strategy](/dbms/lookup-table-usage/index-performance/#index-strategy-lookup)**
- **[PRIMARY KEY Policy](/dbms/lookup-table-usage/primary-key-policy/#policy-lookup-primary-key)**
- **[UPDATE/DELETE with General Predicates](/dbms/lookup-table-usage/predicate-update-delete/)**
- **[Backup and Recovery Support](/dbms/lookup-table-usage/operations-lifecycle/#recovery-support-scope-backup-lookup)**
- **[Limitations and Considerations](/dbms/lookup-table-usage/constraints-errors-troubleshooting/#limitations-lookup)**
---
title: "9.3 Create, Alter, and Drop"
url: https://docs.machbase.com/dbms/lookup-table-usage/create-alter-drop/
language: en
kind: page
---
# 9.3 Create, Alter, and Drop
This section explains how to create, alter, and drop LOOKUP tables.
## Creating and Managing LOOKUP Tables
Create a reference table as follows. A LOOKUP table must specify a `PRIMARY KEY`.
## Creating a LOOKUP Table
```sql
CREATE LOOKUP TABLE ch9_ddl (id INTEGER PRIMARY KEY, name VARCHAR(20));
```
Use meaningful column names for reference data used in production.
```sql
CREATE LOOKUP TABLE ch9_ddl_equip (
equip_id LONG PRIMARY KEY,
equip_name VARCHAR(128),
location VARCHAR(64),
status VARCHAR(16),
updated_at DATETIME
);
```
The `PRIMARY KEY` uniquely identifies rows and provides a basis for UPDATE, DELETE, and JOIN. If the
business key combines several columns, encode that combination in a separate key column or use a
SEQUENCE-based surrogate key.
```sql
CREATE LOOKUP TABLE ch9_ddl_price (
price_key VARCHAR(64) PRIMARY KEY,
product_id VARCHAR(32),
region VARCHAR(16),
price DOUBLE
);
```
## AUTO_INCREMENT PRIMARY KEY
To have the server generate a numeric PRIMARY KEY, specify `AUTO_INCREMENT` on a single `LONG` or
`INT64` column.
```sql
CREATE LOOKUP TABLE ch9_ddl_registry (
equip_id LONG PRIMARY KEY AUTO_INCREMENT,
equip_name VARCHAR(128),
location VARCHAR(64)
);
INSERT INTO ch9_ddl_registry(equip_name, location)
VALUES ('compressor-01', 'SEOUL-A');
```
The server generates a value when the primary key column is omitted or set to NULL. A single
`INSERT ... VALUES` can also specify a primary key in `0..INT64_MAX`. If the specified value is at
least the next automatic value, the next value advances to `specified value + 1`. Smaller values do
not move it backward. Data and the next automatic value survive a normal restart.
LOOKUP tables with AUTO_INCREMENT do not support `INSERT ... SELECT` or `ON DUPLICATE KEY UPDATE`.
For obtaining inserted IDs through an SDK, see
[ROWID and INSERT Result IDs](/dbms/reference/sql/rowid/).
## Using a SEQUENCE Column
For automatic numbering, use a `LONG PROPERTY(SEQUENCE=1)` column. Obtain the next value with
`NEXTVAL()` during insertion.
```sql
CREATE LOOKUP TABLE ch9_ddl_alarm (
seq LONG PROPERTY(SEQUENCE=1) PRIMARY KEY,
sensor_id VARCHAR(64),
alarm_type VARCHAR(32),
occurred_at DATETIME,
message VARCHAR(256)
);
INSERT INTO ch9_ddl_alarm
VALUES (NEXTVAL(seq), 'TEMP-01', 'HIGH', NOW, '온도 초과');
```
For detailed policies, see [SEQUENCE Columns](/dbms/lookup-table-usage/sequence-column/).
`PROPERTY(SEQUENCE)` and `AUTO_INCREMENT` are separate features and must not be specified together
on one column.
## Adding and Dropping Columns
In Standard Edition, you can add and drop fixed-length numeric ARRAY columns in LOOKUP tables.
```sql
ALTER TABLE ch9_ddl_equip
ADD COLUMN (limits DECIMAL(12,4)[2] DEFAULT [0.0000, NULL]);
ALTER TABLE ch9_ddl_equip
DROP COLUMN (limits);
```
Without DEFAULT, the new ARRAY column is a whole-array NULL in existing rows. With DEFAULT, existing
rows also receive that value. The ARRAY DEFAULT must contain exactly the declared number of
elements. ARRAY columns cannot be PRIMARY KEY or index keys.
For supported types and limitations, see [Numeric ARRAY Types](/dbms/reference/sql/types/array/).
## Adding Indexes
Add indexes to columns frequently queried or used in JOIN predicates.
```sql
CREATE INDEX ch9_ddl_loc_idx ON ch9_ddl_equip(location);
CREATE INDEX ch9_ddl_status_idx ON ch9_ddl_equip(status);
```
A default index is created on the PRIMARY KEY column; do not add a duplicate index on it. More
indexes increase insertion and update costs. Add them only for columns with clear query predicates.
## Deleting Data and Dropping Tables
Use `DELETE` to remove rows. A primary key predicate is the clearest approach for a single-row deletion.
```sql
DELETE FROM ch9_ddl_equip
WHERE equip_id = 1001;
```
Bulk deletion can use general predicates. For production data, first check the target count with the
same predicate.
```sql
SELECT COUNT(*)
FROM ch9_ddl_equip
WHERE status = 'RETIRED';
DELETE FROM ch9_ddl_equip
WHERE status = 'RETIRED';
```
Use `DROP TABLE` to remove the table itself.
```sql
DROP TABLE ch9_ddl_alarm;
DROP TABLE ch9_ddl_registry;
DROP TABLE ch9_ddl_price;
DROP TABLE ch9_ddl_equip;
DROP TABLE ch9_ddl;
```
`DROP TABLE` removes both the definition and data. Back up or export data first when needed.
## Considerations
- A `PRIMARY KEY` is required for LOOKUP tables.
- Only one column can be designated as the `PRIMARY KEY`.
- To change a `PRIMARY KEY` value, delete the existing row and insert it with the new key.
- Consider TRANSACTION tables when reference data grows or query/update patterns become complex.
---
title: "9.4 Data Input and Mutation"
url: https://docs.machbase.com/dbms/lookup-table-usage/data-input-mutation/
language: en
kind: page
---
# 9.4 Data Input and Mutation
This section uses one runnable example to explain LOOKUP data insertion, updates, and deletion.
## Prepare the Example Table
Run the following examples in order through the final cleanup statements.
```sql
CREATE LOOKUP TABLE ch9_mutation (
code VARCHAR(32) PRIMARY KEY,
label VARCHAR(64),
status VARCHAR(16),
updated_at DATETIME
);
INSERT INTO ch9_mutation VALUES ('TEMP', 'Temperature', 'ACTIVE', NOW);
INSERT INTO ch9_mutation VALUES ('PRESS', 'Pressure', 'ACTIVE', NOW);
```
If a sequence key is needed, see [SEQUENCE](/dbms/lookup-table-usage/sequence-column/).
## UPDATE
Use a PRIMARY KEY predicate for single-row changes.
```sql
UPDATE ch9_mutation
SET status = 'INACTIVE',
updated_at = NOW
WHERE code = 'TEMP';
```
Before changing multiple rows, query the targets with the same `WHERE` clause. For a single-row
change, a `PRIMARY KEY` predicate is recommended because it identifies the target clearly and can
use an index.
The PRIMARY KEY column itself cannot be updated. To change a key, delete the row and insert it again
with the new key. These statements cannot be grouped into a TRANSACTION table transaction, so also
design for intermediate failures and changes to referencing data.
## Duplicate-Key Handling
Use `ON DUPLICATE KEY UPDATE` when a duplicate key in SQL INSERT should update an existing row.
```sql
INSERT INTO ch9_mutation
VALUES ('TEMP', 'Temperature sensor', 'ACTIVE', NOW)
ON DUPLICATE KEY UPDATE SET label = 'Temperature sensor', status = 'ACTIVE', updated_at = NOW;
```
When Append encounters a duplicate primary key, it can update the row according to
`LOOKUP_APPEND_UPDATE_ON_DUPKEY`. This setting controls duplicate-key handling for the LOOKUP Append
path. Check its current value before using it in production.
```sql
SELECT name, value
FROM v$property
WHERE name = 'LOOKUP_APPEND_UPDATE_ON_DUPKEY';
```
## TABLE_REFRESH
Run `TABLE_REFRESH` when persistently stored LOOKUP content must be reloaded into the running
in-memory table.
```sql
EXEC TABLE_REFRESH(ch9_mutation);
```
Do not run it after every ordinary SQL DML operation. Its target is a LOOKUP table in the current
database, and it cannot run in a READ ONLY database. For name resolution, permissions, and errors,
see the
[EXEC Procedure Reference](/dbms/reference/sql/syntax/execute-procedure-syntax/#table-refresh). For
cluster procedures, see [Operations and Lifecycle](/dbms/lookup-table-usage/operations-lifecycle/).
## Deleting LOOKUP Data
Use a PRIMARY KEY predicate for single-row deletion.
```sql
DELETE FROM ch9_mutation
WHERE code = 'PRESS';
```
A general predicate deletes all matching rows. Omit WHERE to delete every row.
```sql
DELETE FROM ch9_mutation;
DROP TABLE ch9_mutation;
```
## Modification Checklist
- Use PRIMARY KEY predicates for single-row changes.
- Before bulk changes with general predicates, query the target scope using the same predicates.
- Check backups or reload sources before deleting all rows.
- Change PRIMARY KEY values with DELETE followed by INSERT.
- Check `LOOKUP_APPEND_UPDATE_ON_DUPKEY` for Append duplicate-key handling.
- Use `EXEC TABLE_REFRESH(table_name)` only when persistent LOOKUP content must be reloaded into runtime memory.
---
title: "9.5 Query and Analysis"
url: https://docs.machbase.com/dbms/lookup-table-usage/query-analysis/
language: en
kind: page
---
# 9.5 Query and Analysis
This section provides runnable examples of key lookups, general predicate queries, and joins with TAG data.
## Prepare Example Data
Prepare the following LOOKUP and TAG tables.
```sql
CREATE LOOKUP TABLE ch9_query_master (
sensor_id VARCHAR(32) PRIMARY KEY,
site VARCHAR(32),
unit VARCHAR(16),
status VARCHAR(16)
);
INSERT INTO ch9_query_master VALUES ('TEMP-01', 'SEOUL', 'C', 'ACTIVE');
INSERT INTO ch9_query_master VALUES ('TEMP-02', 'BUSAN', 'C', 'INACTIVE');
CREATE TAG TABLE ch9_query_data (
name VARCHAR(32) PRIMARY KEY,
time DATETIME BASETIME,
value DOUBLE SUMMARIZED
);
INSERT INTO ch9_query_data VALUES ('TEMP-01', TO_DATE('2026-01-01 00:00:00'), 23.5);
INSERT INTO ch9_query_data VALUES ('TEMP-02', TO_DATE('2026-01-01 00:00:00'), 19.0);
```
## PRIMARY KEY Lookups
Use a `PRIMARY KEY` predicate for single-row queries.
```sql
SELECT sensor_id, site, unit, status
FROM ch9_query_master
WHERE sensor_id = 'TEMP-01';
```
## General Predicate Queries
LOOKUP tables also support queries on ordinary columns. Add indexes to frequently used predicate columns.
```sql
SELECT sensor_id, site, unit
FROM ch9_query_master
WHERE site = 'SEOUL'
AND status = 'ACTIVE';
```
Indexes can be added to frequently used ordinary predicate columns. For design and creation, see
[Indexes](/dbms/lookup-table-usage/index-performance/).
## Joining TAG and LOG Tables
LOOKUP tables commonly add descriptive information to source data in TAG or LOG tables.
```sql
SELECT d.name, m.site, m.unit, d.time, d.value
FROM ch9_query_data d
JOIN ch9_query_master m ON d.name = m.sensor_id
WHERE m.status = 'ACTIVE';
```
## Analysis Patterns
LOOKUP tables provide analysis criteria rather than storing source data. They suit the following patterns.
| Pattern | Description |
|------|------|
| Code translation | Convert status codes, alarm codes, and device types to labels |
| Reference-value comparison | Join sensor values with thresholds to detect exceedances |
| Grouping criteria | Provide aggregation keys such as location, department, or line |
| Current settings | Apply settings that change during operation at query time |
Similarly, join a threshold LOOKUP table with source TAG data to detect values above a threshold.
Restrict large TAG or LOG time ranges before joining.
```sql
DROP TABLE ch9_query_data CASCADE;
DROP TABLE ch9_query_master;
```
## Query Performance Criteria
- Use `PRIMARY KEY` or indexed columns for single-row lookups and join keys.
- Consider secondary indexes for ordinary columns frequently used in predicates.
- Narrow the source time range before joining large source datasets.
- Store reference data in LOOKUP and long-term source data in TAG or LOG.
---
title: "9.6 Indexes and Performance"
url: https://docs.machbase.com/dbms/lookup-table-usage/index-performance/
language: en
kind: page
---
# 9.6 Indexes and Performance
This section covers LOOKUP index structure and performance tuning.
## Tuning LOOKUP Indexes
A red-black tree index is created automatically for the LOOKUP PRIMARY KEY. All rows and indexes
reside in memory during SQL queries. You can add red-black secondary indexes to non-PK columns when
needed.
### LOOKUP Table Indexes
#### Automatic Primary Key Red-Black Tree Index
Creating a LOOKUP table automatically creates a red-black tree index on its PRIMARY KEY. Each index
entry points to the in-memory row containing all column values for that key. Although the table is
persistent, the query execution path is memory-based and optimized for repeated key lookups on small
reference datasets.
```sql
CREATE LOOKUP TABLE ch9_index_device (
device_id VARCHAR(64) PRIMARY KEY, -- Red-black tree created automatically
device_name VARCHAR(128),
location VARCHAR(256),
category VARCHAR(32)
);
INSERT INTO ch9_index_device VALUES ('DEV-01', 'Boiler', 'Seoul', 'temperature');
INSERT INTO ch9_index_device VALUES ('DEV-02', 'Pump', 'Busan', 'pressure');
```
```sql
-- Primary key lookup using the red-black tree index
SELECT device_id, device_name, location FROM ch9_index_device
WHERE device_id = 'DEV-01';
```
The query returns one row. The LOOKUP side also uses the primary key when joined to event logs.
```sql
CREATE LOG TABLE ch9_index_event (device_id VARCHAR(64), level SHORT);
INSERT INTO ch9_index_event VALUES ('DEV-01', 3);
INSERT INTO ch9_index_event VALUES ('DEV-02', 1);
EXEC TABLE_FLUSH(ch9_index_event);
SELECT e.device_id, e.level, d.location
FROM ch9_index_event e, ch9_index_device d
WHERE e.device_id = d.device_id
ORDER BY e.device_id;
```
The query returns two rows. Adding a time range on the LOG side further reduces source rows read.
#### Secondary Indexes on Non-PK Columns
Red-black secondary indexes can also be created on non-PK columns. Filtering on a column without a
secondary index sequentially scans the entire table.
```sql
-- Add a secondary index to a frequently filtered non-PK column
CREATE INDEX ch9_index_device_location ON ch9_index_device(location);
SELECT device_id FROM ch9_index_device WHERE location = 'Seoul';
-- An unindexed column may require a full scan
SELECT device_id FROM ch9_index_device WHERE category = 'temperature';
```
Both queries return DEV-01. The results match but access paths differ, so compare them with
execution plans on production-scale data.
Secondary indexes speed up queries but increase update cost and memory usage. Create them only on
frequently used predicate columns.
#### LOOKUP Usage Guidelines
Lookup cost for primary key and red-black secondary indexes grows with tree size. Unindexed
predicates scan all memory-resident rows. Do not define capacity limits from row count alone.
Measure actual row sizes, variable-length values, index counts, and query/update ratios under the
same workload. Also check startup time, because startup reads all persistent data to build memory
rows and indexes.
| Usage pattern | Suitability |
|----------|-------|
| Retrieve device information by primary key | Suitable |
| Search lists by non-PK columns | Suitable with secondary indexes |
| Small reference-code tables | Suitable |
| Large reference datasets that do not all fit in server memory | Consider TRANSACTION |
| Reference data requiring relational transactions | Consider TRANSACTION |
### Distinction from VOLATILE
VOLATILE is a separate table type whose data is lost on restart. For its index design, see
[VOLATILE Indexes and Performance](/dbms/volatile-table-usage/index-performance/).
### Large Reference Datasets
Consider alternatives in the following scenario because of LOOKUP table size and secondary-index
update costs.
**Scenario**: Device reference data must be filtered by multiple columns while retaining change history.
```sql
-- Alternative: a LOG table with LSM/BITMAP indexes
CREATE LOG TABLE ch9_index_device_hist (
device_id VARCHAR(64),
device_name VARCHAR(128),
location VARCHAR(256),
category VARCHAR(32),
updated_at DATETIME
);
-- Indexes can be created on non-PK columns
CREATE INDEX ch9_index_hist_location ON ch9_index_device_hist (location);
CREATE INDEX ch9_index_hist_category ON ch9_index_device_hist (category) INDEX_TYPE BITMAP;
```
LOG tables are append-only, so adapt the design to the reference-data update pattern.
Clean up the example objects as follows.
```sql
DROP TABLE ch9_index_device_hist;
DROP INDEX ch9_index_device_location;
DROP TABLE ch9_index_event;
DROP TABLE ch9_index_device;
```
### Key Points
- The PRIMARY KEY index is created automatically.
- Create secondary indexes only for repeated non-PK predicates.
- Measure memory for rows, variable-length values, and indexes, together with startup time and update load.
---
title: "9.7 Operations and Data Lifecycle"
url: https://docs.machbase.com/dbms/lookup-table-usage/operations-lifecycle/
language: en
kind: page
---
# 9.7 Operations and Data Lifecycle
This section covers LOOKUP backup, recovery, and data persistence.
LOOKUP tables persist reference data on disk. Because values can change during operation, manage
change procedures, backups, recovery, and query visibility together.
Distinguish persistent storage from the data location used by queries. At startup, the server
restores all persisted LOOKUP rows into in-memory tables and builds PRIMARY KEY and secondary
indexes. SQL queries use this memory structure during service.
## Data Lifecycle
Manage LOOKUP data through creation, insertion, updates, reference, backup, and recovery.
```
Create table
└── Insert reference data
└── Join or reference in TAG/LOG/TRANSACTION queries
└── UPDATE/DELETE during operation
└── Backup / Recovery / Mount
```
Reference data is smaller than source events but directly affects how query results are interpreted.
Establish procedures to record before/after values and the time changes take effect.
## Reference-Data Change Procedure
Use the following sequence to change LOOKUP data during operation.
1. Query the rows to change.
2. Check the impact scope.
3. Execute UPDATE or DELETE.
4. Execute `EXEC TABLE_REFRESH(table_name)` if needed.
5. Verify visibility with representative queries.
`TABLE_REFRESH` is not required after every ordinary SQL DML operation. Use it when persistent
LOOKUP content must be reloaded into the runtime memory table. For name resolution, permissions, and
errors, see the
[EXEC Procedure Reference](/dbms/reference/sql/syntax/execute-procedure-syntax/#table-refresh).
The following exercise follows this procedure.
```sql
CREATE LOOKUP TABLE ch9_ops_sensor (
sensor_id VARCHAR(32) PRIMARY KEY,
site VARCHAR(16),
status VARCHAR(16),
updated_at DATETIME
);
INSERT INTO ch9_ops_sensor VALUES ('TEMP-01', 'SEOUL', 'READY', NOW);
INSERT INTO ch9_ops_sensor VALUES ('TEMP-02', 'SEOUL', 'READY', NOW);
INSERT INTO ch9_ops_sensor VALUES ('TEMP-03', 'BUSAN', 'READY', NOW);
-- Query the rows to change.
SELECT sensor_id, site, status FROM ch9_ops_sensor WHERE sensor_id = 'TEMP-01';
-- Apply the change.
UPDATE ch9_ops_sensor
SET status = 'INACTIVE', updated_at = NOW
WHERE sensor_id = 'TEMP-01';
-- Reload the memory table if needed.
EXEC TABLE_REFRESH(ch9_ops_sensor);
-- Verify with a representative query.
SELECT sensor_id, status FROM ch9_ops_sensor ORDER BY sensor_id;
```
Only TEMP-01 becomes `INACTIVE`; the other two rows remain `READY`.
Always check the target count before bulk changes.
```sql
SELECT COUNT(*) FROM ch9_ops_sensor
WHERE site = 'SEOUL' AND status = 'READY';
```
COUNT is 1 because TEMP-01 has already changed. Counting after the change gives a different target set.
```sql
DROP TABLE ch9_ops_sensor;
```
## Backup and Recovery Support
LOOKUP data is persisted on disk and included in database backups. After restore, rows are
reconstructed as memory tables and indexes. Use
[Backup, Restore, and Mount](/dbms/operations-configuration-recovery/backup-restore-mount/) as the
authoritative reference for common BACKUP/RESTORE/MOUNT commands and Edition scope. Validate
representative keys and JOIN results after recovery.
## Operational Checks
- Keep reference-data change history in a separate log or operational procedure.
- Check the target count before bulk UPDATE/DELETE.
- Consider indexes on frequent join columns.
- Check startup time and LOOKUP row/index memory usage at actual data scale.
- Check `LOOKUP_APPEND_UPDATE_ON_DUPKEY` when using Append duplicate-key handling.
- Verify reference results with representative JOIN queries after backup recovery.
---
title: "9.8 Constraints, Errors, and Troubleshooting"
url: https://docs.machbase.com/dbms/lookup-table-usage/constraints-errors-troubleshooting/
language: en
kind: page
---
# 9.8 Constraints, Errors, and Troubleshooting
This section covers LOOKUP limitations, possible errors, and troubleshooting.
## Limitation Summary
| Item | Limitation | Typical error |
|---|---|---|
| PRIMARY KEY | Required; only one column | `ERR-02322`, `ERR-02171` |
| PRIMARY KEY column | Cannot be a `SET` target | `ERR-02176` |
| Column types | `TEXT`, `CLOB`, `BLOB`, and `BINARY` unsupported | `ERR-02173` |
| JSON columns | Ordinary columns supported; PRIMARY KEY unsupported | — |
| Memory | Shares one limit with VOLATILE | `ERR-01344` |
| UPDATE | `WHERE` required; does not execute if omitted | — |
## PRIMARY KEY Errors
LOOKUP cannot be created without a PRIMARY KEY or with more than one. Each statement below fails.
```sql
-- Expected failure: no PRIMARY KEY. (ERR-02322)
CREATE LOOKUP TABLE ch9_err_nopk (code VARCHAR(16), label VARCHAR(64));
-- Expected failure: two PRIMARY KEY columns. (ERR-02171)
CREATE LOOKUP TABLE ch9_err_twopk (
code VARCHAR(16) PRIMARY KEY,
name VARCHAR(32) PRIMARY KEY
);
```
Neither statement creates a table, so no cleanup is needed. If a composite key is required, encode
it in one column using a delimiter and follow [PRIMARY KEY Policy](../primary-key-policy/).
PRIMARY KEY column values cannot be updated.
```sql
CREATE LOOKUP TABLE ch9_err_pk (code VARCHAR(16) PRIMARY KEY, label VARCHAR(64));
INSERT INTO ch9_err_pk VALUES ('KR', '대한민국');
-- Expected failure: PRIMARY KEY cannot be a SET target. (ERR-02176)
UPDATE ch9_err_pk SET code = 'KO' WHERE code = 'KR';
```
To change a key, delete the existing row and insert it with the new key.
## Unsupported Column Types
LOOKUP columns cannot use `TEXT`, `CLOB`, `BLOB`, or `BINARY`. Declare long strings as `VARCHAR`;
store raw content separately in LOG when needed.
```sql
-- Expected failure: unsupported column type. (ERR-02173)
ALTER TABLE ch9_err_pk ADD COLUMN (memo TEXT);
```
JSON is supported for ordinary columns. For scope, see [JSON Columns and Queries](../json-column-query/).
```sql
DROP TABLE ch9_err_pk;
```
## Memory Limit
LOOKUP persists data on disk but executes queries in memory. Startup loads all rows and indexes into
memory. Exceeding the limit causes `ERR-01344`.
This limit is **shared with VOLATILE tables**. Although the setting name begins with `VOLATILE_`,
LOOKUP is included. Assess the combined usage when using both types.
```sql
SELECT NAME, VALUE FROM V$PROPERTY
WHERE NAME = 'VOLATILE_TABLESPACE_MEMORY_MAX_SIZE';
SELECT * FROM V$STORAGE_DC_VOLATILE_TABLE;
```
Near the limit, reduce retained data, remove unused secondary indexes, or consider another table
type for large reference datasets using the criteria in
[Indexes and Performance](../index-performance/).
## Scope of Multirow Changes
UPDATE/DELETE with general predicates can affect multiple rows. Check the target count using the
same predicate first and follow
[UPDATE/DELETE with General Predicates](../predicate-update-delete/).
## JSON PRIMARY KEY Errors
JSON can be an ordinary column but cannot be declared as a PRIMARY KEY. Store the identifier in a
separate scalar column and follow the type/path rules in
[JSON Columns and Queries](../json-column-query/).
## Limitations and Considerations
- For primary key rules, see [PRIMARY KEY Policy](../primary-key-policy/).
- Measure memory scale and index cost as described in [Indexes and Performance](../index-performance/).
- Choose TAG for source time series and VOLATILE for caches that may be lost on restart.
- Follow the [SDK Append Matrix](/dbms/development-tools-integration/sdk-support-scope/#append-table-type-matrix) for Append availability.
---
title: "9.9 Patterns and Scenarios"
url: https://docs.machbase.com/dbms/lookup-table-usage/patterns-scenarios/
language: en
kind: page
---
# 9.9 Patterns and Scenarios
This section covers LOOKUP usage patterns and scenarios.
## Use Cases
LOOKUP tables are suitable for the following data.
## Suitable Data Types
| Type | Examples |
|------|------|
| Code tables | Country, language, and status codes |
| Reference data | Equipment lists, product categories, and departments |
| Reference values updated in real time | Exchange-rate tables and threshold settings |
| Alternative to tag metadata | Small sensor-information datasets |
## Code Tables
```sql
CREATE LOOKUP TABLE ch9_pattern_country (
code VARCHAR(4) PRIMARY KEY,
name VARCHAR(64),
region VARCHAR(32)
);
INSERT INTO ch9_pattern_country VALUES ('KR', '대한민국', 'Asia');
INSERT INTO ch9_pattern_country VALUES ('US', '미국', 'America');
UPDATE ch9_pattern_country SET name = 'United States' WHERE code = 'US';
SELECT code, name FROM ch9_pattern_country ORDER BY code;
```
Two rows are returned. Only the US row's name has changed to `United States`.
Manage status and alarm codes the same way and join them to source events.
```sql
CREATE LOOKUP TABLE ch9_pattern_status (
code VARCHAR(16) PRIMARY KEY,
label VARCHAR(64),
color VARCHAR(16)
);
INSERT INTO ch9_pattern_status VALUES ('RUN', '가동', 'green');
INSERT INTO ch9_pattern_status VALUES ('STOP', '정지', 'red');
CREATE LOG TABLE ch9_pattern_event (
event_time DATETIME,
device_id VARCHAR(64),
status VARCHAR(16)
);
INSERT INTO ch9_pattern_event
VALUES (TO_DATE('2026-01-01 10:00:00', 'YYYY-MM-DD HH24:MI:SS'), 'DEV-01', 'RUN');
INSERT INTO ch9_pattern_event
VALUES (TO_DATE('2026-01-01 10:05:00', 'YYYY-MM-DD HH24:MI:SS'), 'DEV-01', 'STOP');
EXEC TABLE_FLUSH(ch9_pattern_event);
SELECT e.event_time, e.device_id, c.label
FROM ch9_pattern_event e
JOIN ch9_pattern_status c ON e.status = c.code
ORDER BY e.event_time;
```
The two rows display the labels `가동` (running) and `정지` (stopped) instead of codes. An event with a
status missing from the code table is excluded by the INNER JOIN, so also check for missing codes.
## Equipment Master Data
```sql
CREATE LOOKUP TABLE ch9_pattern_equip (
equip_id VARCHAR(32) PRIMARY KEY,
equip_name VARCHAR(128),
location VARCHAR(64),
dept VARCHAR(64),
install_dt DATETIME
);
INSERT INTO ch9_pattern_equip
VALUES ('TEMP-01', 'Boiler', 'Seoul', 'Production',
TO_DATE('2025-01-01', 'YYYY-MM-DD'));
INSERT INTO ch9_pattern_equip
VALUES ('TEMP-02', 'Chiller', 'Busan', 'Facility',
TO_DATE('2025-01-01', 'YYYY-MM-DD'));
```
Join sensor data in a TAG table to retrieve reference information such as location and department.
```sql
CREATE TAG TABLE ch9_pattern_sensor (
name VARCHAR(32) PRIMARY KEY,
time DATETIME BASETIME,
value DOUBLE SUMMARIZED
);
INSERT INTO ch9_pattern_sensor
VALUES ('TEMP-01', TO_DATE('2026-01-01 10:00:00', 'YYYY-MM-DD HH24:MI:SS'), 90.0);
INSERT INTO ch9_pattern_sensor
VALUES ('TEMP-02', TO_DATE('2026-01-01 10:00:00', 'YYYY-MM-DD HH24:MI:SS'), 20.0);
EXEC TABLE_FLUSH(ch9_pattern_sensor);
SELECT d.name, m.location, m.dept, d.value
FROM ch9_pattern_sensor d
JOIN ch9_pattern_equip m ON d.name = m.equip_id
WHERE m.dept = 'Production'
ORDER BY d.name;
```
Only TEMP-01 is returned. The department predicate applies to a LOOKUP column, so changing it
changes the result set even when the TAG source data is unchanged.
## Threshold Settings
```sql
CREATE LOOKUP TABLE ch9_pattern_threshold (
sensor_name VARCHAR(64) PRIMARY KEY,
low_limit DOUBLE,
high_limit DOUBLE,
alert_level SHORT
);
INSERT INTO ch9_pattern_threshold VALUES ('TEMP-01', 0.0, 100.0, 1);
INSERT INTO ch9_pattern_threshold VALUES ('TEMP-02', 0.0, 100.0, 1);
-- Change the threshold in real time
UPDATE ch9_pattern_threshold SET high_limit = 85.0 WHERE sensor_name = 'TEMP-01';
```
Join the threshold table with TAG data to evaluate alarm conditions.
```sql
SELECT s.name, s.value, t.high_limit
FROM ch9_pattern_sensor s
JOIN ch9_pattern_threshold t ON s.name = t.sensor_name
WHERE s.value > t.high_limit
ORDER BY s.name;
```
Only TEMP-01 exceeds its threshold. The value 90 was normal under the previous limit of 100. Record
when thresholds change because the same source value can produce a different decision.
## SEQUENCE-Based History Numbers
Use a SEQUENCE column when small administrative histories or operational events need sequential identifiers.
```sql
CREATE LOOKUP TABLE ch9_pattern_note (
seq LONG PROPERTY(SEQUENCE=1) PRIMARY KEY,
target_id VARCHAR(64),
note VARCHAR(512),
created_at DATETIME
);
INSERT INTO ch9_pattern_note
VALUES (NEXTVAL(seq), 'TEMP-01', 'threshold changed', NOW);
INSERT INTO ch9_pattern_note
VALUES (NEXTVAL(seq), 'TEMP-02', 'inspection done', NOW);
SELECT seq, target_id FROM ch9_pattern_note ORDER BY seq;
```
The seq values are 1 and 2. Store large source histories in LOG rather than LOOKUP.
Clean up the example objects as follows.
```sql
DROP TABLE ch9_pattern_note;
DROP TABLE ch9_pattern_threshold;
DROP TABLE ch9_pattern_sensor;
DROP TABLE ch9_pattern_equip;
DROP TABLE ch9_pattern_event;
DROP TABLE ch9_pattern_status;
DROP TABLE ch9_pattern_country;
```
## Unsuitable Cases
- Data requiring explicit transactions and general relational DML → TRANSACTION recommended
- Append-only history requiring no UPDATE → LOG recommended
- Sensor measurements → TAG recommended
- Current-state caches that may be lost on restart → VOLATILE recommended
---
title: "9.10 PRIMARY KEY Policy"
url: https://docs.machbase.com/dbms/lookup-table-usage/primary-key-policy/
language: en
kind: page
---
# 9.10 PRIMARY KEY Policy
This section covers LOOKUP primary key design principles and policies. For how SDKs identify primary
key columns in SELECT results, see
[PRIMARY KEY Metadata Support](/dbms/development-tools-integration/sdk-support-scope/#support-scope-sdk-primary-key-metadata).
## PRIMARY KEY Design
LOOKUP tables require a `PRIMARY KEY`, which uniquely identifies rows and controls duplicates.
UPDATE and DELETE support general predicates in WHERE, but the primary key column itself cannot be
updated.
### Basic Syntax
```text
CREATE LOOKUP TABLE table_name (
pk_col type PRIMARY KEY,
col2 type,
...
);
```
### Single-Column PRIMARY KEY
```sql
CREATE LOOKUP TABLE ch9_pk_country_code (
code VARCHAR(4) PRIMARY KEY,
name VARCHAR(64),
region VARCHAR(32)
);
```
### When a Composite Key Is Needed
```sql
CREATE LOOKUP TABLE ch9_pk_price (
price_key VARCHAR(64) PRIMARY KEY,
product_id VARCHAR(32),
region VARCHAR(16),
price DOUBLE
);
-- Insert
INSERT INTO ch9_pk_price VALUES ('PROD-01:KR', 'PROD-01', 'KR', 99.0);
INSERT INTO ch9_pk_price VALUES ('PROD-01:US', 'PROD-01', 'US', 79.0);
-- Update by composite key
UPDATE ch9_pk_price SET price = 89.0
WHERE price_key = 'PROD-01:KR';
```
Only one LOOKUP column can be designated as the PRIMARY KEY. If a business key combines several
columns, store a combined string or surrogate key in a separate PRIMARY KEY column.
### Choosing a PRIMARY KEY Type
| Type | Advantages | Disadvantages |
|------|------|------|
| `VARCHAR(n)` | Readable, meaningful keys | String comparison cost |
| `INTEGER` / `LONG` | Fast comparison, efficient storage | No business meaning; separate mapping needed |
### Considerations
- PRIMARY KEY values must be unique.
- PRIMARY KEY values cannot be updated; use DELETE + INSERT to change them.
- An index is created automatically on the PRIMARY KEY column.
- Only one column can be designated as the PRIMARY KEY.
## PRIMARY KEY Policy
Consider the following policies and practices when designing primary keys.
### Natural versus Surrogate Keys
#### Natural Key
Use a value with business meaning directly as the PRIMARY KEY.
```sql
-- Country code: a standardized natural key
CREATE LOOKUP TABLE ch9_pk_country (
iso_code VARCHAR(4) PRIMARY KEY, -- ISO 3166-1 alpha-2
name VARCHAR(64)
);
```
**Advantages**: Easy to interpret; no separate lookup required.
**Disadvantages**: Changing keys creates referential-integrity concerns.
#### Surrogate Key
Use a value without business meaning, such as a SEQUENCE column or UUID, as the PRIMARY KEY.
```sql
-- Equipment master: surrogate key
CREATE LOOKUP TABLE ch9_pk_equip (
equip_id LONG PROPERTY(SEQUENCE=1) PRIMARY KEY,
code VARCHAR(32), -- Business key
name VARCHAR(128)
);
CREATE INDEX ch9_pk_equip_idx ON ch9_pk_equip(code);
```
**Advantages**: Immutable; efficient joins.
**Disadvantages**: Code-to-ID mapping required.
### PRIMARY KEY Immutability
PRIMARY KEY values cannot be updated. Use DELETE + INSERT when a change is required.
```sql
-- Incorrect pattern (use DELETE + INSERT for key changes)
-- UPDATE cannot change a primary key
-- Correct pattern
DELETE FROM ch9_pk_country WHERE iso_code = 'OLD';
INSERT INTO ch9_pk_country VALUES ('NEW', '새 국가명');
```
LOOKUP DML executes statement by statement. LOOKUP DML cannot be included in a TRANSACTION table
transaction enclosed by `BEGIN`/`COMMIT`.
Account for reads between the two statements and for insert failure. Retain the old values and
define the transition order for referencing keys before making changes. If the entire change must be
atomic, consider TRANSACTION tables.
Clean up the example tables as follows.
```sql
DROP INDEX ch9_pk_equip_idx;
DROP TABLE ch9_pk_equip;
DROP TABLE ch9_pk_country;
DROP TABLE ch9_pk_price;
DROP TABLE ch9_pk_country_code;
```
---
title: "9.11 SEQUENCE Columns"
url: https://docs.machbase.com/dbms/lookup-table-usage/sequence-column/
language: en
kind: page
---
# 9.11 SEQUENCE Columns
This section covers configuring and using LOOKUP SEQUENCE columns.
## Defining a LOOKUP SEQUENCE Column
A SEQUENCE column generates insertion identifiers through `NEXTVAL`. Because LOOKUP
[requires a PRIMARY KEY](../primary-key-policy/), decide whether the SEQUENCE column or another
column is the primary key. Do not interpret sequence order as event-time order.
## Why Use a SEQUENCE Column?
Use it to distinguish alarms or administrative history rows that share a timestamp. Because LOOKUP
keeps all rows in memory, this example is intended for small administrative histories. Consider LOG
for large event volumes accumulated over long periods.
## Declaring a SEQUENCE Column
SEQUENCE can be specified on `LONG` or `INT64` columns. Set the starting value with the `SEQUENCE`
parameter in PROPERTY. This property is available only for LOOKUP tables.
```sql
CREATE LOOKUP TABLE ch9_sequence (
seq LONG PROPERTY(SEQUENCE=1) PRIMARY KEY,
sensor_id VARCHAR(40),
alarm_type VARCHAR(20),
occurred_at DATETIME,
message VARCHAR(200)
);
```
- `SEQUENCE=1`: automatic numbering for seq starts at 1.
- The starting value must be an integer at least 1 and less than 4,294,967,295.
- Declaring the SEQUENCE column as PRIMARY KEY, as above, also guarantees uniqueness.
## Inserting SEQUENCE Values: NEXTVAL()
The server maintains the next number as a counter for each SEQUENCE column. `NEXTVAL()` obtains that
value for insertion; it does not recalculate the table maximum each time.
```sql
-- Insert an automatically incremented value with NEXTVAL()
INSERT INTO ch9_sequence (seq, sensor_id, alarm_type, occurred_at, message)
VALUES (NEXTVAL(seq), 'TEMP-01', 'HIGH', NOW, '온도 초과');
INSERT INTO ch9_sequence (seq, sensor_id, alarm_type, occurred_at, message)
VALUES (NEXTVAL(seq), 'PRESS-02', 'LOW', NOW, '압력 저하');
-- Query
SELECT * FROM ch9_sequence ORDER BY seq;
-- Sorted as seq=1, then seq=2
```
## Specifying Values Directly
Direct input is also allowed for SEQUENCE columns. If the value exceeds the current counter, the
counter advances to `input value + 1`. A value below the current counter does not move it backward.
```sql
-- Specify a value directly; NEXTVAL is optional
INSERT INTO ch9_sequence (seq, sensor_id, alarm_type, occurred_at, message)
VALUES (100, 'FLOW-03', 'NORMAL', NOW, '정상 복구');
-- The next NEXTVAL() call returns 101
INSERT INTO ch9_sequence (seq, sensor_id, alarm_type, occurred_at, message)
VALUES (NEXTVAL(seq), 'TEMP-01', 'NORMAL', NOW, '온도 정상');
-- seq = 101
```
## Usage Patterns
```sql
-- Query the latest N alarms
SELECT * FROM ch9_sequence ORDER BY seq DESC LIMIT 10;
-- Query alarms after a specified seq
SELECT * FROM ch9_sequence WHERE seq > 500 ORDER BY seq;
-- Acknowledge an alarm with a primary key UPDATE
UPDATE ch9_sequence SET alarm_type = 'ACKNOWLEDGED'
WHERE seq = 101;
```
Clean up the example table as follows.
```sql
DROP TABLE ch9_sequence;
```
## Considerations
- SEQUENCE columns support `LONG` and `INT64`.
- The starting value must be positive (`SEQUENCE=1` or higher) and less than 4,294,967,295.
- The server stores a counter that only increases. Deleting the row with the largest seq does not reuse its number, so gaps can occur.
- Without PRIMARY KEY on the SEQUENCE column, duplicate values can be inserted directly without `NEXTVAL()`. Make it the PRIMARY KEY if uniqueness is required.
---
title: "9.12 JSON Columns and Queries"
url: https://docs.machbase.com/dbms/lookup-table-usage/json-column-query/
language: en
kind: page
---
# 9.12 JSON Columns and Queries
This section covers LOOKUP JSON column support and JSON predicate queries.
## LOOKUP JSON Predicate Queries
LOOKUP supports `JSON` for ordinary columns. Use JSON columns to store flexible attributes alongside
reference data.
```sql
CREATE LOOKUP TABLE ch9_json (
sensor_id VARCHAR(80) PRIMARY KEY,
location VARCHAR(200),
config JSON
);
INSERT INTO ch9_json VALUES (
'TEMP-01',
'factory1',
'{"unit":"celsius","level":3,"threshold":{"high":90.0}}'
);
SELECT sensor_id, config
FROM ch9_json
WHERE config->'$.unit' = 'celsius';
```
## Type-Specific JSON Predicates
Use type-specific JSON extraction functions to compare numeric values as numbers.
```sql
SELECT sensor_id
FROM ch9_json
WHERE JSON_EXTRACT_INTEGER(config, '$.level') >= 3
AND JSON_EXTRACT_DOUBLE(config, '$.threshold.high') > 80.0;
```
You can also inspect the JSON structure itself.
```sql
SELECT sensor_id
FROM ch9_json
WHERE JSON_IS_VALID(config) = 1
AND JSON_TYPEOF(config, '$.threshold') = 'Object';
```
## PRIMARY KEY Restriction
LOOKUP can store JSON columns, but JSON cannot be a `PRIMARY KEY`. Use a stable ordinary type such
as `INTEGER`, `LONG`, or `VARCHAR` for row identifiers.
```sql
-- Expected failure: JSON cannot be a primary key.
CREATE LOOKUP TABLE ch9_json_bad (
config JSON PRIMARY KEY,
note VARCHAR(80)
);
```
```sql
-- Recommended: use a separate identifier as the primary key.
CREATE LOOKUP TABLE ch9_json_ok (
sensor_id VARCHAR(80) PRIMARY KEY,
config JSON,
note VARCHAR(80)
);
```
## Design Criteria
| Situation | Recommended approach |
|------|----------|
| Values frequently used in joins/searches | Separate columns |
| Flexible attributes that differ by device | JSON column |
| Numeric predicate queries | Extract into ordinary numeric columns |
| Primary key | Stable identifier column |
| Frequent path queries | Extract into separate columns |
Dedicated JSON path indexes are unsupported. For frequent predicates, first consider extracting the
values into separate indexed columns.
```sql
CREATE LOOKUP TABLE ch9_json_fast (
sensor_id VARCHAR(80) PRIMARY KEY,
unit VARCHAR(16),
level INTEGER,
config JSON
);
CREATE INDEX ch9_json_unit_idx ON ch9_json_fast(unit);
```
## UPDATE and DELETE Predicates
```sql
UPDATE ch9_json
SET location = 'factory2'
WHERE config->'$.unit' = 'celsius';
DELETE FROM ch9_json
WHERE JSON_EXTRACT_INTEGER(config, '$.level') < 2;
```
Because the target scope can be broad, check the count with the same predicate before UPDATE/DELETE.
Clean up the example objects as follows. `ch9_json_bad` is excluded because its creation intentionally fails.
```sql
DROP TABLE ch9_json_fast;
DROP TABLE ch9_json_ok;
DROP TABLE ch9_json;
```
## Considerations
- LOOKUP supports JSON as an ordinary column.
- JSON cannot be a primary key.
- Dedicated JSON path indexes are unsupported.
- Extract frequently searched values into ordinary LOOKUP columns.
- Consider TRANSACTION or TAG tables if JSON path indexes are required.
---
title: "9.13 Predicate UPDATE/DELETE"
url: https://docs.machbase.com/dbms/lookup-table-usage/predicate-update-delete/
language: en
kind: page
---
# 9.13 Predicate UPDATE/DELETE
LOOKUP tables can update or delete multiple rows using general predicates as well as primary keys.
Check the target row count with the same predicate before making changes.
This exercise uses one table, which is cleaned up at the end.
```sql
CREATE LOOKUP TABLE ch9_predicate (
equip_id VARCHAR(32) PRIMARY KEY,
site VARCHAR(16),
status VARCHAR(16),
score INTEGER
);
INSERT INTO ch9_predicate VALUES ('EQ-01', 'SEOUL', 'READY', 10);
INSERT INTO ch9_predicate VALUES ('EQ-02', 'SEOUL', 'READY', 20);
INSERT INTO ch9_predicate VALUES ('EQ-03', 'SEOUL', 'RETIRED', 30);
INSERT INTO ch9_predicate VALUES ('EQ-04', 'BUSAN', 'READY', 40);
```
## UPDATE
All matching rows are updated. `SET` expressions can reference current row values, but the primary
key column itself cannot be changed.
LOOKUP UPDATE requires `WHERE`. Specify a supported predicate even when updating every row. This
differs from DELETE, which permits deleting all rows without WHERE.
```sql
-- Count affected rows before the change.
SELECT COUNT(*) FROM ch9_predicate
WHERE site = 'SEOUL' AND status = 'READY';
UPDATE ch9_predicate
SET status = 'ACTIVE', score = score + 10
WHERE site = 'SEOUL' AND status = 'READY';
SELECT equip_id, site, status, score FROM ch9_predicate ORDER BY equip_id;
```
COUNT is 2. Only EQ-01 and EQ-02 become `ACTIVE`, with scores of 20 and 30. EQ-03 remains unchanged
because its status differs, despite being in SEOUL; EQ-04 remains unchanged because its site
differs, despite being READY.
## DELETE
All matching rows are deleted. Omitting `WHERE` deletes every row in the table.
```sql
SELECT COUNT(*) FROM ch9_predicate WHERE status = 'RETIRED';
DELETE FROM ch9_predicate WHERE status = 'RETIRED';
SELECT equip_id, status FROM ch9_predicate ORDER BY equip_id;
```
COUNT is 1. EQ-03 is deleted, leaving three rows.
## Predicate Design Guidelines
1. Use primary key predicates for single-row changes.
2. Check the impact scope with `SELECT COUNT(*)` using the same predicate before bulk changes.
3. Consider extracting frequently filtered JSON values into ordinary indexed columns.
4. To change a primary key, delete the existing row and insert it with the new key.
Check the SQL reference for the precise scope of supported operators and JSON predicates.
- [LOOKUP predicate UPDATE](/dbms/reference/sql/syntax/dml-syntax/lookup-predicate-update-syntax/)
- [LOOKUP predicate DELETE](/dbms/reference/sql/syntax/dml-syntax/lookup-predicate-delete-syntax/)
## Permissions and Performance
UPDATE and DELETE require the respective `UPDATE` and `DELETE` privileges on the target table. Grant
`SELECT` only if the application itself reads before/after values. Use
[Privilege Management](/dbms/security-access-control/privileges/) as the authoritative reference for
privilege SQL.
Primary key equality directly locates one row; general predicates evaluate conditions to collect
target rows. Use prepared statements and binding for repeated single-row changes. For bulk changes,
measure the row count and execution time for the same predicate in a validation environment.
```sql
DROP TABLE ch9_predicate;
```
---
title: "10. VOLATILE Table Usage"
url: https://docs.machbase.com/dbms/volatile-table-usage/
language: en
kind: section
---
# 10. VOLATILE Table Usage
VOLATILE tables are in-memory tables shared across the server process. This chapter covers data loss
on restart, UPSERT, and cache patterns that can be rebuilt.
## Chapter Contents
| Section | Topics |
|----|------|
| [Overview and Use Criteria](./overview-use-criteria/) | Characteristics and selection criteria |
| [Table Structure and Schema](./table-structure-schema/) | PRIMARY KEY design, column types, and schema structure |
| [Create, Alter, and Drop](./create-alter-drop/) | CREATE VOLATILE TABLE, DROP, and persistence differences |
| [Data Ingestion and Modification](./data-input-mutation/) | INSERT, ON DUPLICATE KEY UPDATE, and DELETE |
| [Queries and Analysis](./query-analysis/) | SELECT, predicate queries, and LIKE |
| [Indexes and Performance](./index-performance/) | Red-black tree indexes and primary key indexes |
| [Operations and Data Lifecycle](./operations-lifecycle/) | Operational procedures and data management |
| [Constraints, Errors, and Troubleshooting](./constraints-errors-troubleshooting/) | Feature limitations and error handling |
---
title: "10.1 Overview and Use Criteria"
url: https://docs.machbase.com/dbms/volatile-table-usage/overview-use-criteria/
language: en
kind: page
---
# 10.1 Overview and Use Criteria
VOLATILE tables store temporary data in memory. Data is lost when the server restarts, so use them
for current state that can be rebuilt, temporary aggregates, and caches shared across sessions.
## VOLATILE Table Characteristics
Create a VOLATILE table with `CREATE VOLATILE TABLE`.
```sql
CREATE VOLATILE TABLE ch10_overview (
sensor_id VARCHAR(64) PRIMARY KEY,
value DOUBLE,
updated_at DATETIME
);
```
VOLATILE tables have the following characteristics.
| Item | Description |
|------|------|
| Main uses | Current-state caches, temporary aggregates, and intermediate results |
| Storage | Memory |
| Data after restart | Lost |
| Sharing scope | Server-wide |
| Key | Optional PRIMARY KEY |
| Main features | UPDATE, DELETE, `ON DUPLICATE KEY UPDATE`, and red-black tree indexes |
| Backup | Not supported |
## Use Criteria
Use a VOLATILE table when:
- Data may be lost after a server restart.
- Data can be recalculated or rebuilt from its source at any time.
- Current state, recent aggregates, or temporary results require fast queries.
- Multiple sessions need to share the same temporary state.
- In-memory response time is more important than disk persistence.
The following example maintains the latest sensor state.
```sql
INSERT INTO ch10_overview VALUES ('TEMP-01', 23.5, NOW)
ON DUPLICATE KEY UPDATE SET value = 23.5, updated_at = NOW;
SELECT *
FROM ch10_overview
WHERE sensor_id = 'TEMP-01';
-- Clean up because the next section reuses this name.
DROP TABLE ch10_overview;
```
### Usage Patterns
| Pattern | Key | Source for rebuilding | Recommended expiration method |
|------|-----|-------------|----------------|
| Latest device state | Device ID | TAG or LOG | Update the same key |
| Short-interval aggregates | Target and time bucket | TAG or LOG | Replace or rebuild the bucket |
| Job progress | Job ID | Job system | Delete the key after completion |
| Temporary query cache | Request or object ID | Persistent table | Rebuild the entire cache |
For current-state updates, see [Data Ingestion and Modification](../data-input-mutation/). For
temporary aggregates, see [Queries and Analysis](../query-analysis/).
## When to Consider Other Tables
Use another table type for the following requirements.
| Requirement | Recommended table |
|----------|-------------|
| Source data that must survive a restart | TAG or LOG |
| Persistent reference data, such as code lists or device master data | LOOKUP |
| Business data requiring transactions and relational updates | TRANSACTION |
| Time-series data for long-term analysis | TAG |
Data stored only in a VOLATILE table cannot be recovered after server shutdown. Store important data
in a suitable persistent table: TAG, LOG, LOOKUP, or TRANSACTION. Use VOLATILE tables for caches or
intermediate results.
## Design Sequence
Make the following decisions when designing a VOLATILE table.
1. Confirm that the data can be rebuilt.
2. Decide whether a PRIMARY KEY is required.
3. Estimate the row count and memory usage.
4. Prepare an initial load procedure for use after restart.
5. Save results that must be retained to persistent tables through explicit application writes.
There is no dedicated flush command that persists a VOLATILE table.
For schema and primary key design, see
[Table Structure and Schema](/dbms/volatile-table-usage/table-structure-schema/). For restart
handling, see [Restart and Data Loss](/dbms/volatile-table-usage/operations-lifecycle/).
---
title: "10.2 Table Structure and Schema"
url: https://docs.machbase.com/dbms/volatile-table-usage/table-structure-schema/
language: en
kind: page
---
# 10.2 Table Structure and Schema
This section covers primary key design and schema structure for VOLATILE tables.
## PRIMARY KEY Design
You can create a VOLATILE table without a PRIMARY KEY. However, a PRIMARY KEY is required for
primary key lookups and `ON DUPLICATE KEY UPDATE`.
### Single PRIMARY KEY
```sql
CREATE VOLATILE TABLE ch10_schema_state (
device_id VARCHAR(32) PRIMARY KEY,
state VARCHAR(16),
updated_at DATETIME
);
```
### When a Composite Key Is Needed
If a combination of columns uniquely identifies a row, encode that combination in a separate PRIMARY
KEY column.
```sql
CREATE VOLATILE TABLE ch10_schema_hourly (
key_id VARCHAR(96) PRIMARY KEY,
sensor_id VARCHAR(64),
hour_ts DATETIME,
avg_value DOUBLE,
sample_cnt INTEGER
);
-- Insert composite key values
INSERT INTO ch10_schema_hourly
VALUES ('TEMP-01:2026010110', 'TEMP-01', TO_DATE('2026-01-01 10:00:00', 'YYYY-MM-DD HH24:MI:SS'), 23.5, 60);
INSERT INTO ch10_schema_hourly
VALUES ('TEMP-01:2026010111', 'TEMP-01', TO_DATE('2026-01-01 11:00:00', 'YYYY-MM-DD HH24:MI:SS'), 24.1, 60);
INSERT INTO ch10_schema_hourly
VALUES ('TEMP-02:2026010110', 'TEMP-02', TO_DATE('2026-01-01 10:00:00', 'YYYY-MM-DD HH24:MI:SS'), 21.0, 60);
-- Query by composite key
SELECT sensor_id, avg_value FROM ch10_schema_hourly
WHERE key_id = 'TEMP-01:2026010110';
```
### Using ON DUPLICATE KEY UPDATE
```sql
-- Update when the PRIMARY KEY already exists
INSERT INTO ch10_schema_state VALUES ('DEV-01', 'ONLINE', NOW)
ON DUPLICATE KEY UPDATE SET state = 'ONLINE', updated_at = NOW;
```
### Considerations
- A table can be created without a PRIMARY KEY, but primary key operations such as UPSERT and primary key lookups are unavailable.
- Multiple rows cannot have the same PRIMARY KEY. `ON DUPLICATE KEY UPDATE` updates the existing row instead of adding a duplicate.
- Only one column can be designated as the PRIMARY KEY.
## VOLATILE Table Design
VOLATILE tables keep data only in memory and lose it on server restart. The table definition
remains. Decide which data can be lost and how to reload it. Use these tables for state or caches
shared by multiple sessions that do not need to survive a restart.
Review the following topics when defining the schema.
- [Use Cases](../overview-use-criteria/#use-cases-volatile)
- [Persistence Differences and DDL](../create-alter-drop/#differences-persistence-ddl)
- [Memory Lifecycle](../operations-lifecycle/#lifecycle-memory)
- [Red-Black Tree Indexes](../index-performance/#index-strategy-red-black)
- [ON DUPLICATE KEY UPDATE](../data-input-mutation/#on-duplicate-key-update)
- [Restart and Data Rebuilding](../operations-lifecycle/#data-loss)
Clean up the example tables as follows.
```sql
DROP TABLE ch10_schema_hourly;
DROP TABLE ch10_schema_state;
```
---
title: "10.3 Create, Alter, and Drop"
url: https://docs.machbase.com/dbms/volatile-table-usage/create-alter-drop/
language: en
kind: page
---
# 10.3 Create, Alter, and Drop
This section explains how to create and drop VOLATILE tables and how their persistence differs from
other table types.
## Creating and Managing VOLATILE Tables
Use the following statements to create and drop a VOLATILE table.
### Create
```sql
create volatile table vtable (id1 integer, name varchar(20));
```
### Drop
```sql
drop table vtable;
```
## Persistence Differences and DDL
Unlike other table types, VOLATILE tables store data only in memory.
### Persistence Comparison
| Item | VOLATILE | LOOKUP | TRANSACTION | TAG | LOG |
|------|----------|--------|-----|-----|-----|
| Storage | Memory | Disk | Disk | Disk | Disk |
| Data retained after server restart | No | Yes | Yes | Yes | Yes |
| Table structure (DDL) retained | Yes | Yes | Yes | Yes | Yes |
### DDL Behavior
Only the data in a VOLATILE table is lost. Its definition is stored persistently, as with other
table types. After a restart, perform an initial load; the table does not need to be recreated.
```sql
-- Create the table definition only once.
CREATE VOLATILE TABLE ch10_ddl (
sensor_id VARCHAR(64) PRIMARY KEY,
value DOUBLE,
updated_at DATETIME
);
```
### Creation Syntax
The basic syntax is `CREATE VOLATILE TABLE table_name (column_definition, ...)`. Designate one
column as `PRIMARY KEY` when key-based updates or deletes are required.
- `PRIMARY KEY` is optional.
- Only one column can be designated as the PRIMARY KEY.
### AUTO_INCREMENT PRIMARY KEY
To have the server generate a numeric PRIMARY KEY, specify `AUTO_INCREMENT` on a single `LONG` or
`INT64` column.
```sql
CREATE VOLATILE TABLE ch10_ddl_seq (
request_id LONG PRIMARY KEY AUTO_INCREMENT,
payload VARCHAR(256)
);
INSERT INTO ch10_ddl_seq(payload) VALUES('refresh');
```
The server generates a value when the primary key column is omitted or set to NULL. A single
`INSERT ... VALUES` statement can also specify a primary key value in the range `0..INT64_MAX`. If
the specified value is at least the next automatic value, the next automatic value advances to
`specified value + 1`. A smaller value does not move it backward.
VOLATILE tables with AUTO_INCREMENT do not support `INSERT ... SELECT` or `ON DUPLICATE KEY UPDATE`.
Because VOLATILE table data is lost on server restart, the next automatic value also restarts at 1.
The table definition remains and does not need to be recreated. For obtaining the inserted ID
through an SDK, see [ROWID and INSERT Result IDs](/dbms/reference/sql/rowid/).
### Adding and Dropping Columns
In Standard Edition, you can add and drop fixed-length numeric ARRAY columns in VOLATILE tables.
```sql
ALTER TABLE ch10_ddl
ADD COLUMN (thresholds DOUBLE[2] DEFAULT [10.0, 20.0]);
ALTER TABLE ch10_ddl
DROP COLUMN (thresholds);
```
As with scalar `ADD COLUMN`, VOLATILE does not backfill rows that existed before ALTER with the
DEFAULT value. Even with an ARRAY DEFAULT, the new column is a whole-array NULL in existing rows.
This differs from the backfill rules for LOG, LOOKUP, TRANSACTION, and TAG METADATA.
For supported ARRAY element types, cardinality, and DEFAULT rules, see
[Numeric ARRAY Types](/dbms/reference/sql/types/array/).
### Drop
```sql
DROP TABLE ch10_ddl;
DROP TABLE ch10_ddl_seq;
```
### Considerations
- VOLATILE table DDL is stored in the database and survives server restart.
- Data is not restored automatically after restart. Configure an initial load script, for example by running machsql at startup.
---
title: "10.4 Data Input and Mutation"
url: https://docs.machbase.com/dbms/volatile-table-usage/data-input-mutation/
language: en
kind: page
---
# 10.4 Data Input and Mutation
This section provides runnable examples of `INSERT`, duplicate-key updates, and `DELETE` for VOLATILE tables.
## Inserting and Updating Data
Run the following examples in order through the final cleanup statement. To repeatedly change values
for the same key, such as current state, define a `PRIMARY KEY` and use `ON DUPLICATE KEY UPDATE`.
```sql
CREATE VOLATILE TABLE ch10_mutation (
id INTEGER PRIMARY KEY,
direction VARCHAR(10),
refcnt INTEGER
);
INSERT INTO ch10_mutation VALUES (1, 'west', 0);
INSERT INTO ch10_mutation VALUES (2, 'east', 0);
INSERT INTO ch10_mutation VALUES (1, 'south', 0)
ON DUPLICATE KEY UPDATE;
INSERT INTO ch10_mutation VALUES (1, 'south', 0)
ON DUPLICATE KEY UPDATE SET refcnt = 1;
SELECT * FROM ch10_mutation ORDER BY id;
```
If the key does not exist, a new row is inserted. If it exists, the statement without `SET` updates
the entire row with the input values; the statement with `SET` updates only the specified columns.
The `PRIMARY KEY` itself cannot be updated.
Bulk ingestion APIs differ by language and driver in initialization, binding, and error handling.
Use the corresponding driver example in [SDKs and Integration](/dbms/development-tools-integration/)
instead of copying incomplete snippets.
## Conditional Updates
Use `UPDATE` to change selected columns of an existing row. `INSERT ... ON DUPLICATE KEY UPDATE`
inserts a missing row or updates an existing one; `UPDATE` changes values only when the target row
exists.
`UPDATE` on a VOLATILE table requires `WHERE` and supports only a `PRIMARY KEY = value` predicate,
as with conditional deletion. Predicates on other columns and compound predicates are unsupported.
Updating all rows by omitting `WHERE` is also unsupported.
```sql
UPDATE ch10_mutation SET refcnt = refcnt + 1 WHERE id = 2;
SELECT * FROM ch10_mutation ORDER BY id;
```
The `SET` clause cannot specify the `PRIMARY KEY` column. To change a key, delete the existing row
and insert it again with the new key.
## Deleting Data
Conditional deletion supports only a `PRIMARY KEY = value` predicate. Predicates on other columns
and compound predicates are unsupported.
```sql
DELETE FROM ch10_mutation WHERE id = 2;
SELECT * FROM ch10_mutation ORDER BY id;
DROP TABLE ch10_mutation;
```
To remove all rows while retaining the table definition, omit WHERE, as in `DELETE FROM table_name`.
To reset the schema as well, drop and recreate the table. A server restart removes only the data and
preserves the definition, so maintain a separate reload script rather than a recreation script.
---
title: "10.5 Query and Analysis"
url: https://docs.machbase.com/dbms/volatile-table-usage/query-analysis/
language: en
kind: page
---
# 10.5 Query and Analysis
This section provides runnable examples of key lookups, general predicate queries, and temporary
aggregates for VOLATILE tables.
## Prepare Example Data
Query VOLATILE tables with `SELECT`, as with other table types. Run the following examples in order
through the final cleanup statements.
```sql
CREATE VOLATILE TABLE ch10_query (
device_id VARCHAR(64) PRIMARY KEY,
status VARCHAR(16),
value DOUBLE,
updated_at DATETIME
);
INSERT INTO ch10_query VALUES ('DEV-01', 'RUNNING', 42.5, NOW);
INSERT INTO ch10_query VALUES ('DEV-02', 'STOPPED', 0, NOW);
```
## PRIMARY KEY Lookups
A key predicate is suitable for retrieving a single row from a current-state cache.
```sql
SELECT device_id, status, value, updated_at
FROM ch10_query
WHERE device_id = 'DEV-01';
```
## General Predicate Queries
Consider a secondary index for repeated queries on columns other than the `PRIMARY KEY`.
```sql
CREATE INDEX ch10_query_status_idx ON ch10_query(status);
SELECT device_id, value, updated_at
FROM ch10_query
WHERE status = 'RUNNING';
```
Indexes also consume memory. Create them only on columns needed by actual queries.
## Querying Temporary Aggregates
Storing aggregates for short intervals reduces repeated calculations for dashboards and alarm evaluation.
```sql
CREATE VOLATILE TABLE ch10_query_summary (
summary_key VARCHAR(96) PRIMARY KEY,
sensor_id VARCHAR(64),
bucket_time DATETIME,
avg_value DOUBLE,
max_value DOUBLE,
sample_cnt LONG
);
INSERT INTO ch10_query_summary
VALUES ('TEMP-01:2026-01-01T00:00', 'TEMP-01', TO_DATE('2026-01-01 00:00:00'),
21.5, 23.0, 60);
SELECT sensor_id, bucket_time, avg_value, max_value
FROM ch10_query_summary
WHERE sensor_id = 'TEMP-01'
ORDER BY bucket_time DESC
LIMIT 10;
DROP TABLE ch10_query_summary;
DROP TABLE ch10_query;
```
If aggregate results require long-term retention, copy them periodically to a LOG or TRANSACTION table.
## Query Considerations
- Data is empty after a server restart. First check whether the initial load has completed.
- Define a `PRIMARY KEY` when key lookups are the primary access pattern.
- Consider secondary indexes for columns frequently used in range queries or sorting.
- Store important source data in persistent tables and use VOLATILE as a cache.
---
title: "10.6 Indexes and Performance"
url: https://docs.machbase.com/dbms/volatile-table-usage/index-performance/
language: en
kind: page
---
# 10.6 Indexes and Performance
This section explains how to create and choose indexes for VOLATILE tables.
## Supported Indexes
Declaring a `PRIMARY KEY` creates an index for key lookups. You can add `REDBLACK` indexes to
ordinary columns. VOLATILE tables do not support `BITMAP` or `KEYWORD` indexes.
Run the following example in order from table creation through cleanup.
```sql
CREATE VOLATILE TABLE ch10_index (
id INTEGER PRIMARY KEY,
name VARCHAR(20),
status VARCHAR(16)
);
CREATE INDEX ch10_index_name_idx
ON ch10_index(name) INDEX_TYPE REDBLACK;
INSERT INTO ch10_index VALUES (1, 'west device', 'ACTIVE');
INSERT INTO ch10_index VALUES (2, 'east device', 'INACTIVE');
SELECT id, name
FROM ch10_index
WHERE name = 'west device';
DROP INDEX ch10_index_name_idx;
DROP TABLE ch10_index;
```
## Design Criteria
- Use a `PRIMARY KEY` for single-row key lookups and updates.
- Add secondary indexes only for repeated equality or range predicates on ordinary columns.
- Indexes consume memory as well as data. Remove unnecessary indexes.
- Compare response time and memory usage before and after creating indexes, using actual query predicates and row counts.
For syntax details, see [Index Syntax](/dbms/reference/sql/syntax/index-syntax/).
---
title: "10.7 Operations and Data Lifecycle"
url: https://docs.machbase.com/dbms/volatile-table-usage/operations-lifecycle/
language: en
kind: page
---
# 10.7 Operations and Data Lifecycle
This section describes the procedures for creating, loading, using, discarding, and rebuilding
VOLATILE table data.
## Data Lifecycle
1. Run the table creation SQL after the server starts.
2. Load initial data from a persistent source if needed.
3. Start application queries and updates.
4. Write results that must be retained to persistent tables.
5. Data is lost when the server shuts down. Table definitions remain.
## Sharing Across Sessions
VOLATILE tables are shared server-wide. A row inserted by one session can be queried by another.
Closing a connection alone does not remove data.
## Separating Temporary and Persistent Data
Store only current state or intermediate results that can be rebuilt from source data in VOLATILE.
Store audit records, source events, and results that cannot be recreated in TAG, LOG, LOOKUP, or
TRANSACTION tables. Manage copy SQL as a separate job, including source and target columns,
duplicate handling, and execution frequency.
## Restart Procedure
- Check whether the table exists. A restart preserves its definition, so recreation is usually unnecessary.
- If a persistent source exists, load only data for the defined reference point in time.
- Check the expected row count and latest timestamp.
- Resume collector and application writes after validation.
- If rebuilding fails, confirm that the service operates safely with an empty cache.
## Operations Checklist
- Keep initial load SQL in version control, together with creation SQL for the first deployment.
- Validate scripts in advance using the operational account and actual connection settings.
- Monitor row counts and memory limits.
- Check that data requiring retention is not stored only in VOLATILE.
- Verify the loading and validation sequence during restart drills.
## Checking Memory and Rebuilding Caches
```sql
SELECT * FROM V$STORAGE_DC_VOLATILE_TABLE;
SELECT * FROM V$SYSMEM;
SELECT * FROM V$SESMEM;
SELECT NAME, VALUE
FROM V$PROPERTY
WHERE NAME = 'VOLATILE_TABLESPACE_MEMORY_MAX_SIZE';
```
Check the view definitions for your deployed version instead of relying on specific internal column
names. When rebuilding a cache, record the row count and sample values, redirect its consumers, and
then create the table, load initial data, and validate it in that order. The service must be able to
operate safely with an empty cache if rebuilding fails.
---
title: "10.8 Constraints, Errors, and Troubleshooting"
url: https://docs.machbase.com/dbms/volatile-table-usage/constraints-errors-troubleshooting/
language: en
kind: page
---
# 10.8 Constraints, Errors, and Troubleshooting
This section covers VOLATILE table limitations, possible errors, and troubleshooting. Most issues
involve memory limits, primary key design, unsupported column types, or data loss after restart.
## Limitations
Consider the following limitations when using VOLATILE tables.
| Item | Limitation |
|------|------|
| Storage | Memory |
| Data after restart | Lost |
| Backup and mount | Not supported |
| JSON columns | Not supported |
| PRIMARY KEY | Optional; only one column |
| UPDATE/DELETE | Only `PRIMARY KEY = value` predicates are supported |
| Memory limit | Subject to the total memory limit for Volatile/Lookup tables |
```sql
-- Expected failure: VOLATILE tables do not support JSON columns.
CREATE VOLATILE TABLE ch10_err_json (
session_id VARCHAR(64) PRIMARY KEY,
payload JSON
);
```
For flexible attributes, extract frequently queried values into ordinary columns. If persistent JSON
columns are needed, consider TRANSACTION or TAG tables.
## Insufficient Memory
VOLATILE table data and indexes consume memory. Increasing row counts or too many indexes can
exhaust the memory limit.
Diagnose the issue in the following order. The example tables are cleaned up at the end of this page.
```sql
-- Create a table for the diagnostic example.
CREATE VOLATILE TABLE ch10_diag (
device_id VARCHAR(64) PRIMARY KEY,
value DOUBLE
);
INSERT INTO ch10_diag VALUES ('DEV-01', 10.0);
-- 1. Check the target table row count.
SELECT COUNT(*) FROM ch10_diag;
```
```sql
-- 2. Check memory usage across VOLATILE tables.
SELECT *
FROM V$STORAGE_DC_VOLATILE_TABLE;
```
If needed, check the `VOLATILE_TABLESPACE_MEMORY_MAX_SIZE` setting.
```sql
SELECT NAME, VALUE
FROM V$PROPERTY
WHERE NAME = 'VOLATILE_TABLESPACE_MEMORY_MAX_SIZE';
```
To resolve the issue:
- Delete unnecessary rows.
- Reduce the amount of data retained in the cache.
- Remove unused indexes.
- Move important data to persistent tables before rebuilding the VOLATILE table.
- Consider adjusting the memory limit according to operational policy.
## PRIMARY KEY Errors
A PRIMARY KEY is required for `ON DUPLICATE KEY UPDATE`,
[primary key UPDATE](../data-input-mutation/#volatile-primary-key-update), and primary key DELETE.
```sql
CREATE VOLATILE TABLE ch10_err_device (
device_id VARCHAR(64) PRIMARY KEY,
status VARCHAR(16),
updated_at DATETIME
);
```
PRIMARY KEY values must be unique. Use `ON DUPLICATE KEY UPDATE` to treat duplicate inserts as updates.
```sql
INSERT INTO ch10_err_device VALUES ('DEV-01', 'ONLINE', NOW)
ON DUPLICATE KEY UPDATE SET status = 'ONLINE', updated_at = NOW;
```
The PRIMARY KEY column itself cannot be updated. To change a key, delete the existing row and insert
it with the new key.
## Data Loss After Restart
VOLATILE table data is lost on normal shutdown, abnormal termination, or restart. This is a
characteristic of the table type, not an error.
When an issue occurs, check the following.
1. Check whether the server has restarted.
2. Check whether the VOLATILE table creation script was executed.
3. Check whether the initial load query completed successfully.
4. Rebuild the cache from the source TAG/LOG/TRANSACTION table.
```sql
SELECT COUNT(*) FROM ch10_diag;
```
If the result is 0, run the initial load procedure again.
Clean up the example tables as follows.
```sql
DROP TABLE ch10_diag;
DROP TABLE ch10_err_device;
```
## Troubleshooting Checklist
- Confirm that the stored data can be rebuilt.
- Check whether the operation requires a PRIMARY KEY.
- Use `COUNT(*)` and `V$STORAGE_DC_VOLATILE_TABLE` to check size and memory usage.
- Run the initial load SQL again after restart. The table does not need to be recreated.
- For persistent retention, use TAG, LOG, LOOKUP, or TRANSACTION tables instead of VOLATILE.
---
title: "11. Development and Application Integration"
url: https://docs.machbase.com/dbms/development-tools-integration/
language: en
kind: section
---
# 11. Development and Application Integration
Choose an integration method for your application, then review SDK installation, APIs,
and shared operational principles. Use SDK pages for language-specific implementations
and the selection, concepts, and support pages for guidance shared across SDKs.
## Reading Order
1. [Choose an Integration Method](selection-integration-method/) for your language and input pattern.
2. Review authentication, time values, binding, transactions, and retries in
[Common Integration Concepts](concepts-common/).
3. Compare actual support in [SDK Feature Support](sdk-support-scope/).
4. Follow the relevant SDK page for installation, connections, and executable code.
5. Apply task-specific procedures in [Data Ingestion and Export](data-input-load-export/).
For SQL ROWID semantics, see [ROWID](../reference/sql/rowid/). For partial ARRAY
input in Machbase DBMS 8.7.0, see
[Sparse ARRAY and Selected-Column Append API](data-input-load-export/array-append/).
## SDK References
| Environment | Documentation |
|---|---|
| Native C/C++ or ODBC | [Machbase SQLCLI and ODBC](cli-odbc/) |
| Java/Spring | [JDBC](jdbc/) |
| Python | [Python](python/) |
| Node.js/TypeScript | [Node.js / TypeScript](node-js-typescript/) |
| C#/VB.NET | [.NET Connector](net-connector/) |
| Native Go/`database/sql` | [Go](go/) |
## Common Connection Information
| Item | Default | Description |
|---|---|---|
| HOST | `127.0.0.1` | Server host name or IP address |
| PORT | `5656` | Server port (`PORT_NO` in `machbase.conf`) |
| USER | `SYS` | User ID |
| PASSWORD | `MANAGER` | User password |
Use a dedicated user with the minimum required privileges in production. See
[Accounts, Privileges, and Access Control](../security-access-control/) for account
and authentication settings.
## Existing Support Links
See [SDK Feature Support](sdk-support-scope/) for NULL/PRIMARY KEY metadata, ROWID,
Append, and AUTH KEY support by SDK.
---
title: "11.1 Choose an Integration Method"
url: https://docs.machbase.com/dbms/development-tools-integration/selection-integration-method/
language: en
kind: section
---
# 11.1 Choose an Integration Method
Choose an integration method for the project language, ingestion pattern, and deployment environment.
## Selection Criteria
| Requirement | First option to consider |
|----------|------------------|
| Native C/C++ collector | SQLCLI |
| ODBC manager/DSN application | ODBC |
| Java/Spring | JDBC |
| Python analytics/automation | `machbaseapi` |
| C#/VB.NET | .NET Connector |
| Go collector/service | Native `machgo` or `database/sql` |
| Node.js/TypeScript backend | `@machbase/ts-client` |
| R analytics | Machbase ODBC driver with RODBC |
| Bulk file ingestion/export | machloader, csvimport, csvexport |
| Continuous bulk TAG/LOG ingestion | The chosen SDK's Append API |
Do not connect a browser directly to port 5656. Execute queries in a backend and return
only the required results.
## Decision Sequence
1. Choose an official driver maintainable in the application language.
2. Check port 5656 connectivity, operating system support, and runtime compatibility.
3. Identify required SQL, prepared statement, Append, and transaction features.
4. Check actual support in the [SDK Feature Matrix](../sdk-support-scope/).
5. Round-trip sample timestamps, NULLs, numbers, and strings.
6. Load-test target row sizes, concurrent connections, and batch sizes.
Do not choose a driver on Append support alone. Verify flush latency, server error
responses, reconnection, and failed-row handling in the actual SDK. For TRANSACTION
DML, check explicit transaction support. Do not assume TAG/LOG Append shares its rollback scope.
## Detailed Documentation by Topic
| Topic | Detailed documentation |
|------|------|
| SDK installation, connections, APIs, complete code | SDK pages in this chapter |
| Shared authentication, binding, transactions, retries | [Common Integration Concepts](../concepts-common/) |
| SDK feature support | [SDK Feature Support](../sdk-support-scope/) |
| SQL, configuration, command-line details | [Chapter 16 Reference](/dbms/reference/) |
| Choose ingestion/export methods | [Data Ingestion and Export](../data-input-load-export/) |
After choosing an integration method, run its SDK installation and connection examples.
Check both documented versions and actual deployment artifacts for feature support.
---
title: "11.2 Common Integration Concepts"
url: https://docs.machbase.com/dbms/development-tools-integration/concepts-common/
language: en
kind: section
---
# 11.2 Common Integration Concepts
This page covers connection, binding, transaction, bulk ingestion, and error-handling
principles shared across drivers and languages. See the SDK pages for function names
and complete code.
## Connection Strings and Authentication
Connections require a host, native port, user, and authentication information. The
default port is `5656`; check `machbase.conf` in the deployed environment.
| Item | Check |
|------|-----------|
| Host/port | TCP reachability from the application host |
| User | Minimum privileges for the target database and tables |
| Password | Supply through environment variables or a secret manager |
| Database | SDK support for initial database selection |
| Timeouts | Connection, command, and read limits appropriate for the workload |
| Time zone | Supported option names and scope in the SDK and server |
Example `SYS`/`MANAGER` credentials are for local validation. Create dedicated accounts
for production applications; do not record passwords in source, command history, or logs.
AUTH KEY signs challenges with a private key instead of a password. Check key formats,
file permissions, and SDK options in
[AUTH KEY Authentication](/dbms/security-access-control/authentication-auth-key/)
and the driver documentation.
With connection pools, verify reset behavior so a returned connection's database,
session settings, and open statements do not affect the next request.
## Time Zones and Time Values
Machbase `DATETIME` supports nanosecond precision. Manage the meaning of a timestamp
separately from its representation.
- Specify the time reference for collected data: UTC or the business time zone.
- Fix both format and time zone when binding strings.
- Check whether the SDK expects epoch seconds, milliseconds, microseconds, or nanoseconds.
- Round-trip output time zones through the actual connection options or `TO_CHAR()` path.
- Do not interchange `NOW` and `SYSDATE` in business rules; check their meanings in
the SQL reference.
For string round-trip tests, compare input, query output, and output after a time zone
change on the same connection. See [SQL Functions](/dbms/reference/sql/functions/functions-full/)
for function details.
## Prepared statement
Prepared statements separate SQL structure from values and support repeated execution
of the same SQL.
```text
INSERT INTO sensor_data VALUES (?, ?, ?)
SELECT value FROM sensor_data WHERE name = ? AND time >= ?
```
Check that the preparing connection and current database have not changed. Close the
statement after use. For SDK statement caches, check cache scope, eviction policy,
and reset behavior on database changes.
Some syntax positions, such as CTEs or LIMIT, may not allow parameter placeholders.
On a syntax error, check [Named Bind Parameter](/dbms/reference/sql/syntax/named-bind-parameter-syntax/)
and SDK placeholder support before concatenating values into SQL.
## Parameter binding
| Value | Recommended method |
|---------|-----------|
| Integers/floating-point numbers | Match fixed-width language types to SQL ranges |
| Strings | Check encoding and maximum length |
| DATETIME | Use SDK time objects or the specified epoch unit |
| NULL | Specify the language NULL representation and SQL type |
| DECIMAL | Prefer the SDK exact fixed-point type over string conversion |
| Binary/IP | Use SDK-required byte arrays or dedicated types |
Positional `?` placeholders bind in occurrence order. Use named `:name` placeholders
only with supported servers and SDKs, and check repeated-name rules. Identifiers and
SQL keywords cannot be bound as values; validate them against an allowlist before
constructing SQL.
## DML Affected-Row Counts
After `INSERT`, `UPDATE`, or `DELETE`, check the SDK affected-row count. A success
response alone does not prove the intended business record changed.
- For a single-row change, check that the count is 1.
- For 0 rows, check whether no row matched or the change was already applied.
- Check error codes and exceptions separately for insufficient privileges or unsupported DML.
- Before bulk changes, check the scope with `COUNT(*)` using the same condition.
- For Append, check success/failure counts from close/server responses instead of SQL affected rows.
## Transactions
Use explicit `BEGIN`, `COMMIT`, and `ROLLBACK` for relational DML on TRANSACTION tables.
Do not assume LOG/TAG Append shares the same rollback scope.
1. Check whether the SDK provides a transaction API.
2. If not, execute supported transaction-control SQL on the same connection.
3. Verify rollback and connection reuse on error paths.
4. Leave no unfinished transaction when returning a pooled connection.
5. For mixed table types, verify each statement's commit scope in advance.
See [TRANSACTION Table Transactions](/dbms/rdb-table-usage/transaction/) for complete SQL examples.
## Append APIs and Batches
See [Data Ingestion and Export](../data-input-load-export/) for input choices and result
checks, and the [SDK Append Matrix](../sdk-support-scope/#append-table-type-matrix) for
client/table-type support. Separate Append and ordinary query connections, and check
flush, close, and failed rows.
## Error Handling and Retries
Classify errors as connection, authentication/authorization, SQL/schema, data, or resource exhaustion.
- Retry only disconnects and transient timeouts, with bounded attempts and backoff.
- Do not automatically retry authentication, privilege, syntax, or type errors before fixing them.
- Clean up statements, cursors, Append handles, and connections before retrying.
- Make INSERT retries idempotent with business keys or duplicate-handling policies.
- Log server error codes and messages, excluding credentials and raw sensitive data.
- Validate failed pooled connections before returning them, or discard them.
See [Troubleshooting](/dbms/troubleshooting/) for operational error classification and diagnosis.
---
title: "11.3 SDK Feature Support"
url: https://docs.machbase.com/dbms/development-tools-integration/sdk-support-scope/
language: en
kind: section
---
# 11.3 SDK Feature Support
Compare feature support and API entry points to choose an SDK for your application.
See each SDK page for installation, connections, functions, and runnable examples.
If choosing an SDK for the first time, read [Choose an Integration Method](../selection-integration-method/)
first, then use this page to compare required features and exact API paths.
## Nullable Metadata
Machbase reports SELECT result column nullability as `NO_NULLS`, `NULLABLE`, or `UNKNOWN`.
Applications must handle NULL for both `NULLABLE` and `UNKNOWN`.
| SDK | Retrieval method |
|---|---|
| JDBC | `ResultSetMetaData.isNullable()` |
| Python | `cursor.description[i][6]` |
| Go native | `api.Column.Nullability` |
| Go `database/sql` | `Rows.ColumnTypeNullable()` |
| Node.js | `ColumnMeta.nullable` |
| .NET | `AllowDBNull` in `GetSchemaTable()` |
| SQLCLI/ODBC | Descriptor nullable attribute |
Expressions, aggregates, VIEWs, and JOIN results may report `UNKNOWN`. Do not equate
a source column schema constraint with query result metadata.
## PRIMARY KEY Metadata
| SDK | Result columns | Table catalog |
|---|:---:|:---:|
| JDBC | O | `DatabaseMetaData.getPrimaryKeys()` |
| Python | O | Catalog SQL |
| Go native | O | Catalog SQL |
| Go `database/sql` | No standard API | Catalog SQL |
| Node.js | O | Catalog SQL |
| .NET | O | Catalog SQL |
| ODBC | No separate result API | `SQLPrimaryKeys()` |
Expressions, aggregates, and the NULL-supplying side of an outer join may not preserve
the source column primary key attribute.
## ROWID from INSERT
In Machbase 8.7.0 Standard Edition, a successful single `INSERT ... VALUES` can provide
a ROWID to supported SDKs.
| SDK/tool | Retrieval method | When absent |
|---|---|---|
| machsql | `SHOW LAST ROWID` | `NULL` |
| Machbase SQLCLI | `SQLGetGeneratedRowID()` | `SQL_NO_DATA` |
| Standard ODBC | No dedicated standard API | - |
| JDBC | `Statement.getGeneratedKeys()` | Empty `ResultSet` |
| Python | `cursor.lastrowid` | `None` |
| .NET | `MachCommand.RowId` | `null` |
| Go `database/sql` | `Result.LastInsertId()` | Error |
| Go native | Unsupported | - |
| Node.js | Execution result `rowId` | `undefined` |
Batches, `executemany()`, Append, loaders, `INSERT ... SELECT`, and UPSERT do not return
a single ROWID. See [ROWID](/dbms/reference/sql/rowid/) for SQL semantics and table-specific
constraints.
## Append APIs and Table Types
| API path | LOG | TAG | LOOKUP | VOLATILE | TRANSACTION | Basis |
|---|:---:|:---:|:---:|:---:|:---:|---|
| SQLCLI `SQLAppend*` extension | O | O | O | O | O | TRANSACTION requires Standard |
| JDBC `MachStatement.executeAppend*` | O | O | O | O | O | NFX cce422d source/tests |
| Python 2.4 `append*` | O | O | O | O | O | NFX cce422d source/tests |
| .NET `MachAppendWriter` | O | O | O | O | O | NFX cce422d provider |
| Go v1.8.4 native `Appender` | O | O | X | X | O | TRANSACTION requires Standard |
| Go `database/sql` standard API | X | X | X | X | X | `sql.Conn.Raw()` extensions follow the native contract |
| Node source `appendBatch()` | O | △ | △ | △ | O | NFX cce422d source build |
| Node source `appendOpen()` | O | O | △ | △ | △ | Generic native/fallback path; table-specific validation required |
Keep ordinary query connections and Append handle lifecycles distinct, and check close
and flush results. `O` indicates support verified in the cited source/tests. `△` means
only a generic path is available and further table-specific regression testing is needed.
Append extensions are not standard ODBC or `database/sql` features.
### ARRAY and Selected-Column Append
Machbase DBMS 8.7.0 ARRAY support is as follows:
| SDK | Dense ARRAY retrieval/input | Sparse ARRAY | Selected-column Append |
|---|:---:|:---:|:---:|
| SQLCLI/C++ | O | O | O |
| Machbase ODBC extension | O | O | O |
| JDBC | O | O | O |
| Python | O | O | O |
| Node.js | O | O | O |
| .NET full/legacy provider | O | O | O |
| Go | v2 main source | v2 main source | v2 main source |
Use a Machbase DBMS 8.7.0 server with an SDK build that includes ARRAY support. SQL ARRAY
element positions and Machbase-specific SDK positions are 0-based. Existing full-row
scalar Append APIs are unchanged. The Node.js prepared fallback also supports `SparseArray`.
Go requires v2 main source after [`neo-client` PR #17](https://github.com/machbase/neo-client/pull/17);
until a public v2 release is specified, do not assume a published module version provides
support. See [Sparse ARRAY and Selected-Column Append API](../data-input-load-export/array-append/)
for input methods and APIs.
## AUTH KEY
| SDK/tool | Support |
|---|:---:|
| machsql, Machbase SQLCLI, ODBC, JDBC | O |
| Go native, Go `database/sql` | O (neo-client v1.5.0+) |
| Python, Node.js, .NET | X |
See [AUTH KEY Authentication](/dbms/security-access-control/authentication-auth-key/) for
key generation, registration, and rotation. See supported SDK pages for connection options.
## Transactions, Prepare, and Binding
| SDK | Transaction API | Server prepared | Named bind API |
|---|:---:|:---:|:---:|
| JDBC | O | O | △ (Machbase extension) |
| Python | X | O | O |
| Go native | △ | O | O |
| Go `database/sql` | O | O | O |
| .NET | X | X | △ |
| Node.js | X | O | O |
| SQLCLI | △ | O | O |
| ODBC | △ | O | Positional binding |
Prepared statements and parameter binding are separate from transaction support. See
[Named Bind Parameter](/dbms/reference/sql/syntax/named-bind-parameter-syntax/) for placeholder syntax.
For TAG data UPDATE in Machbase 8.7.0 Standard Edition, existing positional/named SDK
APIs can bind NAME and BASETIME predicate values. NFX #4127 regression tests cover C/C++
SQLCLI, Go `database/sql`, JDBC, Node.js, Python, and .NET. ODBC is not included in that
SDK regression matrix; it binds `?` or named SQL placeholders by ordinal through standard
`SQLBindParameter()`. See
[TAG Data UPDATE Binding](/dbms/reference/sql/syntax/dml-syntax/tag-data-update-syntax/#tag-data-update-predicate-bind)
for predicate requirements.
`△` in the Transaction column means executing transaction SQL on the same connection
instead of using a dedicated object. `△` in Named bind means a driver extension or a
client path that converts values to SQL literals, rather than a standard named binding API.
## Verified Versions and Provenance
| SDK | Verification basis |
|---|---|
| SQLCLI/JDBC/Python | NFX `cce422d2972`; Python package 2.4 |
| Node.js | NFX `cce422d2972` source build (`package.json` 1.0.1) |
| .NET | Uni 8.0.55, limited 3.1.3, full 3.2.2 |
| Go | Released neo-client v1.8.4; AUTH KEY v1.5.0+, database selection v1.8.3+ |
Some Node features in NFX cce422d were added after the public npm 1.0.1 release. Do not
assume feature parity from the registry package version string. Check the artifact
commit provenance or use an NFX source build.
ARRAY and selected-column Append were verified against NFX
`655d1333870313c4951698b89c9a3c9ada11d630` and merge commit
`f756986c4836982723e2aa7727ec05b7e05e9707`. Use Machbase DBMS 8.7.0 as the server baseline
and verified SDK artifacts containing those changes as the client baseline.
## SDK References
| SDK | Detailed documentation |
|---|---|
| Machbase SQLCLI/ODBC | [SQLCLI and ODBC](../cli-odbc/) |
| JDBC | [JDBC](../jdbc/) |
| Python | [Python](../python/) |
| Node.js / TypeScript | [Node.js / TypeScript](../node-js-typescript/) |
| .NET Connector | [.NET Connector](../net-connector/) |
| Go | [Go](../go/) |
---
title: "11.4 Machbase SQLCLI and ODBC"
url: https://docs.machbase.com/dbms/development-tools-integration/cli-odbc/
language: en
kind: section
---
# 11.4 Machbase SQLCLI and ODBC
Machbase SQLCLI is a Call-Level Interface for C/C++ applications. The ODBC driver serves
standard ODBC applications. Both interfaces share an execution flow based on environment,
connection, and statement handles. SQLCLI adds extension functions for high-speed Append.
## Choose an Interface
| Requirement | Interface |
|----------|------------|
| Develop C/C++ applications with the Machbase installation package | SQLCLI |
| Use generic ODBC tools or a driver manager | ODBC |
| Bulk ingestion through Append extensions | SQLCLI |
| Execute standard SQL and retrieve results | SQLCLI or ODBC |
## Headers and Libraries
Check the following files in the installation directory:
```bash
test -f "$MACHBASE_HOME/include/machbase_sqlcli.h"
test -f "$MACHBASE_HOME/lib/libmachbasecli_dll.so"
```
Example dynamic linking on Linux:
```bash
gcc cli_quickstart.c -I"$MACHBASE_HOME/include" -L"$MACHBASE_HOME/lib" -lmachbasecli_dll -o cli_quickstart
LD_LIBRARY_PATH="$MACHBASE_HOME/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" MACHBASE_PASSWORD='your-password' ./cli_quickstart
```
Base production builds on the installation package's `install/machbase_env.mk` and
platform-specific linker settings.
## Connection Strings
The basic SQLCLI connection string uses these keys:
```text
SERVER=127.0.0.1;PORT_NO=5656;UID=APP_USER;PWD=secret;CONNTYPE=1
```
For an ODBC data source, specify the DSN, user, and password.
```text
DSN=MACHBASE;UID=APP_USER;PWD=secret
```
Drivers that support an initial database for multi-database operation accept `DATABASE`
or `DBNAME`. Check support in the deployed driver, then run `SELECT CURRENT_DATABASE()`
after connecting to verify the selection.
## Quick Start
This program connects to port 5656, queries a system table, and releases all handles.
The password is supplied through an environment variable.
```c
#include
#include
#include
int main(void)
{
SQLHENV env = SQL_NULL_HENV;
SQLHDBC dbc = SQL_NULL_HDBC;
SQLHSTMT stmt = SQL_NULL_HSTMT;
char conn[512];
const char *password = getenv("MACHBASE_PASSWORD");
if (password == NULL) {
fputs("MACHBASE_PASSWORD is required\n", stderr);
return 2;
}
snprintf(conn, sizeof(conn),
"SERVER=127.0.0.1;PORT_NO=5656;"
"UID=SYS;PWD=%s;CONNTYPE=1", password);
if (SQLAllocEnv(&env) != SQL_SUCCESS) {
return 3;
}
if (SQLAllocConnect(env, &dbc) != SQL_SUCCESS) {
SQLFreeEnv(env);
return 4;
}
if (SQLDriverConnect(
dbc, NULL, (SQLCHAR *)conn, SQL_NTS,
NULL, 0, NULL, SQL_DRIVER_NOPROMPT) != SQL_SUCCESS) {
SQLFreeConnect(dbc);
SQLFreeEnv(env);
return 5;
}
if (SQLAllocStmt(dbc, &stmt) != SQL_SUCCESS) {
SQLDisconnect(dbc);
SQLFreeConnect(dbc);
SQLFreeEnv(env);
return 6;
}
if (SQLExecDirect(
stmt, (SQLCHAR *)"SELECT COUNT(*) FROM V$TABLES",
SQL_NTS) != SQL_SUCCESS) {
SQLFreeStmt(stmt, SQL_DROP);
SQLDisconnect(dbc);
SQLFreeConnect(dbc);
SQLFreeEnv(env);
return 7;
}
puts("query succeeded");
SQLFreeStmt(stmt, SQL_DROP);
SQLDisconnect(dbc);
SQLFreeConnect(dbc);
SQLFreeEnv(env);
return 0;
}
```
To report failures, read SQLSTATE, the native error code, and the message using
`SQLGetDiagRec()` or `SQLError()` in existing code.
## Standard Execution Flow
1. Allocate environment and connection handles.
2. Connect with `SQLDriverConnect()` or `SQLConnect()`.
3. Allocate a statement handle.
4. Execute SQL with `SQLPrepare()` and `SQLExecute()`, or `SQLExecDirect()`.
5. Read SELECT results with `SQLBindCol()` and `SQLFetch()`.
6. Release statement, connection, and environment resources in that order.
Bind input with `SQLBindParameter()` instead of concatenating strings. Check nullability
with the last argument of `SQLDescribeCol()` or with
`SQLColAttribute(..., SQL_DESC_NULLABLE, ...)`.
## Named Bind Parameter
If the server and driver support named parameters, use `:name` placeholders and
`SQLBindParameterByName()`. A repeated name binds one value to all matching positions. See
[Named Bind Parameter](/dbms/reference/sql/syntax/named-bind-parameter-syntax/) for common
constraints and examples.
When support is unverified, use standard `?` placeholders and `SQLBindParameter()`.
## ROWID from INSERT
To obtain the generated ROWID after a successful single `INSERT ... VALUES` in Standard
Edition, use the following:
- SQLCLI extension: `SQLGetGeneratedRowID()`
- Standard ODBC: no standard API dedicated to generated ROWID
Do not assume the same return behavior for batches, Append, `INSERT ... SELECT`, or UPSERT.
See [ROWID and INSERT Result IDs](/dbms/reference/sql/rowid/) for the exact scope.
## Append Extensions
High-speed ingestion uses an Append flow separate from ordinary statements.
| Step | Main functions |
|------|-----------|
| Open | `SQLAppendOpen()`; `SQLAppendOpenColumns()`/`W()` for selected columns |
| Send a row | `SQLAppendDataV2()` or an Append function supported by the version |
| Send a batch | `SQLAppendBatch()` |
| Flush to the server | `SQLAppendFlush()` |
| Error callback | `SQLAppendSetErrorCallback()` |
| Close | `SQLAppendClose()` |
Append column order and types must match the target schema exactly. Represent strings,
binary data, IP addresses, DATETIME, and NULL according to `SQL_APPEND_PARAM` in the
installed `machbase_sqlcli.h`. Record failed rows and server errors in the callback,
without logging passwords or raw sensitive data.
Do not share a connection with an active Append handle with ordinary queries. Check
success and failure counts returned by close.
## Threads and Resource Management
- Use separate connections and statements per thread.
- Do not use a statement or Append handle concurrently from multiple threads.
- Provide cleanup routines that release handles in reverse order on every error path.
- Before retrying, ensure the previous connection and Append state are fully closed.
- Record both success and failure counts for bulk ingestion.
## Check API Details
The installed `$MACHBASE_HOME/include/machbase_sqlcli.h` is the reference for function
prototypes, constants, and structures matching the library. Do not mix examples with
headers from another version. Include compilation, linking, and a port 5656 connection
test in the deployment pipeline.
## DECIMAL Append
For `DECIMAL` or `NUMERIC` input through `SQLAppendDataV2()` and `SQLAppendBatch()`, use
the 32-byte opaque `SQL_APPEND_NUMERIC` type and its public constructors. Do not create
or modify its internal bytes in application code.
| Input | Function |
|---|---|
| UTF-8 numeric string | `SQLAppendNumericFromString()` |
| Signed/unsigned integer | `SQLAppendNumericFromInt64()`, `SQLAppendNumericFromUInt64()` |
| `SQL_NUMERIC_STRUCT` | `SQLAppendNumericFromSQLNumeric()` |
| NULL | `SQLAppendNumericSetNull()` |
Prefer strings or `SQL_NUMERIC_STRUCT` to preserve exact values. Specify
`SQL_APPEND_TYPE_NUMERIC` or `SQL_APPEND_TYPE_DECIMAL` in the type array. Check overflow
and rounding against the target column precision and scale.
## ARRAY and Selected-Column Append
Machbase DBMS 8.7.0 supports typed ARRAY retrieval and binding with
`SQL_MACHBASE_ARRAY_DESC`, and sparse input with `SQL_MACHBASE_SPARSE_ARRAY_DESC`.
Standard Open can also pass a sparse descriptor to an ARRAY column.
```c
SQLAppendOpen(statement, (SQLCHAR *)"ARRAY_APPEND_FULL_EXAMPLE", 0);
row[0].mLong = 1;
row[1].mVar.mData = &sparse;
row[1].mVar.mLength = SQL_APPEND_SPARSE_ARRAY_DESC_LENGTH;
SQLAppendDataV3(statement, row, 2);
SQLAppendClose(statement, &success, &failure);
```
This code follows the input order of a table with `ID LONG, A INT32[4]`. Start with the
[complete standard Open example](../data-input-load-export/array-append/#c-full-open),
including connection, descriptor and buffer setup, and error handling. This differs
from passing descriptors to the legacy `SQLAppendData(void *[])` API.
To select specific columns or fixed ARRAY elements, use `SQLAppendOpenColumns()` or
`SQLAppendOpenColumnsW()`.
```c
SQLCHAR *targets[] = {
(SQLCHAR *)"ID",
(SQLCHAR *)"CHANNELS[0]",
(SQLCHAR *)"CHANNELS[3]",
NULL
};
SQLAppendOpenColumns(statement, (SQLCHAR *)"SENSOR_ARRAY", targets, 0);
```
ARRAY element targets and sparse descriptor positions are 0-based. The column-name
list must end with `NULL`. `SQLAppendBatch()` does not support ARRAY. See
[Sparse ARRAY and Selected-Column Append API](../data-input-load-export/array-append/)
for descriptor definitions, whole-array and element NULL handling, and direct ODBC
handle restrictions.
---
title: "11.5 JDBC"
url: https://docs.machbase.com/dbms/development-tools-integration/jdbc/
language: en
kind: section
---
# 11.5 JDBC
The Machbase JDBC driver provides core JDBC 4.2 APIs with a Java 8 baseline. Use standard
JDBC APIs for connections, PreparedStatements, typed retrieval and binding, database
metadata, local transactions, and connection pools.
| Item | Value |
|------|----|
| Java bytecode baseline | Java 8 |
| Reported JDBC version | 4.2 |
| Driver version | 3.0.0 |
| JDBC URL | `jdbc:machbase:///[database]` |
| `Driver.jdbcCompliant()` | `false` |
A `false` result from `jdbcCompliant()` concerns full SQL-92 Entry Level compliance,
not JDBC 4.2 API support. Check required optional features through `DatabaseMetaData`
capability methods.
## Multiple Databases
Specify the initial database in the URL path or the `database` connection property.
```java
String url = "jdbc:machbase://127.0.0.1:5656/factory_a";
Connection conn = DriverManager.getConnection(url, "APP_A", password);
System.out.println(conn.getCatalog());
conn.setCatalog("FACTORY_A");
```
`getCatalog()` and `setCatalog()` synchronize with the server's current database. If both
the URL path and property specify a database, their values must match. In JDBC metadata,
a catalog is a database and a schema is an owner. Check that pooled connections restore
the initial catalog when returned. Prepared statements and append handles remain bound
to the database in which they were created.
## Install the Driver
### Use a JAR File
Add `machbase.jar` from the Machbase installation directory to the classpath.
```bash
ls -l "$MACHBASE_HOME/lib/machbase.jar"
javac -classpath ".:$MACHBASE_HOME/lib/machbase.jar" MyApp.java
java -classpath ".:$MACHBASE_HOME/lib/machbase.jar" MyApp
```
The JAR includes `META-INF/services/java.sql.Driver`. In JDBC 4.0 and later environments,
the driver registers automatically without
`Class.forName("com.machbase.jdbc.MachDriver")`. Existing explicit calls remain valid.
### Maven
```xml
com.machbasemachjdbc{{< jdbc_version >}}
```
### Gradle
```groovy
dependencies {
implementation 'com.machbase:machjdbc:{{< jdbc_version >}}'
}
```
Check artifact versions in [Maven Central](https://mvnrepository.com/artifact/com.machbase/machjdbc).
The runtime metadata version `3.0.0` and distribution artifact versions use different
versioning schemes.
## Connect to the Server
Supply user names and passwords through environment variables or a secret manager
instead of embedding them in source code.
```java
import java.sql.Connection;
import java.sql.DriverManager;
import java.util.Properties;
String url = "jdbc:machbase://127.0.0.1:5656/machbasedb";
Properties properties = new Properties();
properties.setProperty("user", "SYS");
properties.setProperty("password", System.getenv("MACHBASE_PASSWORD"));
try (Connection connection =
DriverManager.getConnection(url, properties)) {
// Execute SQL.
}
```
### Connection Options
Specify connection options through `Properties` or URL query parameters. Set
`randomHost` in `Properties`, or use `^` separators in a multi-host URL.
| Option | Description |
|------|------|
| `user`, `password` | Password authentication credentials |
| `TIMEZONE` | Session time zone in `+0900` format |
| `randomHost` | Selects the first connection host randomly from the list |
| `maxStatements` | Maximum cached Statements for a pooled connection |
| `CONNECTION_TIMEOUT` | Socket connection timeout in seconds; `0` means unlimited |
| `SOCKET_TIMEOUT` | Socket read timeout in seconds; `0` means unlimited |
| `characterEncoding` | Client character encoding |
| `AUTH_MODE` | `PASSWORD` or `CHALLENGE` |
| `AUTH_SIG_SCHEME` | `ECDSA`, `RSA_PKCS1_V15`, `RSA_PSS` |
| `AUTH_KEY_FILE` | Path to a PEM private key file |
```java
String url =
"jdbc:machbase://127.0.0.1:5656/machbasedb?TIMEZONE=+0900";
```
### Multiple Hosts
The Machbase 8.7.0 JDBC driver accepts multiple hosts in one URL.
| Selection | Syntax | Behavior |
|-----------|-----------|------|
| Sequential | Separate hosts with `,` | Attempts connections in URL order |
| Random start | Separate hosts with `^` | Randomly selects the first host |
| Random start | Set `randomHost=true` in `Properties` | Randomly selects the first host from a comma-separated list |
This URL tries `db2` if connecting to `db1` fails:
```java
String url =
"jdbc:machbase://db1.example.com:5656,db2.example.com:5656/" +
"machbasedb?CONNECTION_TIMEOUT=5";
```
The `^` separator selects the first host randomly:
```java
String url =
"jdbc:machbase://db1.example.com:5656^db2.example.com:5656/" +
"machbasedb?CONNECTION_TIMEOUT=5";
```
When using the `randomHost` property, separate hosts with commas:
```java
Properties properties = new Properties();
properties.setProperty("randomHost", "true");
String url =
"jdbc:machbase://db1.example.com:5656,db2.example.com:5656/" +
"machbasedb?CONNECTION_TIMEOUT=5";
```
- Do not mix `,` and `^` separators in one URL.
- Connection-stage I/O errors, such as connection refusal, connection timeout, or socket
errors, cause an attempt to the next host. If all hosts fail,
`DriverManager.getConnection()` throws `SQLException`.
- `CONNECTION_TIMEOUT` applies per host attempt. Total connection wait time can therefore
increase with the number of hosts and their response times.
- `SOCKET_TIMEOUT` limits reads on the connected socket; it does not change host selection order.
Multi-host failover applies to socket establishment for new connections or reconnections.
Even if automatic reconnection succeeds after a disconnect, do not reuse earlier Statements,
PreparedStatements, or ResultSets. It does not guarantee whether in-flight SQL succeeded
or can be safely retried. On a connection error in an active transaction, discard the
connection and retry the entire transaction according to the application idempotency policy.
## AUTH KEY Authentication
Public-key challenge authentication signs the server challenge with a local private
key instead of using a password.
```java
Properties properties = new Properties();
properties.setProperty("user", "app_user");
properties.setProperty("AUTH_MODE", "CHALLENGE");
properties.setProperty("AUTH_SIG_SCHEME", "ECDSA");
properties.setProperty(
"AUTH_KEY_FILE", "/opt/machbase/keys/app_user_ecdsa.pem");
Connection connection = DriverManager.getConnection(
"jdbc:machbase://127.0.0.1:5656/machbasedb", properties);
```
- `AUTH_MODE=CHALLENGE` does not use `password` for authentication.
- `AUTH_KEY_FILE` is required.
- If `AUTH_SIG_SCHEME` is omitted, a default scheme is selected for the key type.
- On POSIX systems, restrict private key file permissions to `600`.
## Quick Start
This example inserts values into a LOG table and queries them.
```java
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.Statement;
import java.util.Properties;
public class JdbcQuickStart {
public static void main(String[] args) throws Exception {
Properties properties = new Properties();
properties.setProperty("user", "SYS");
properties.setProperty(
"password", System.getenv("MACHBASE_PASSWORD"));
try (Connection connection = DriverManager.getConnection(
"jdbc:machbase://127.0.0.1:5656/machbasedb",
properties);
Statement statement = connection.createStatement()) {
statement.execute(
"CREATE LOG TABLE jdbc_sensor " +
"(ts DATETIME, name VARCHAR(40), value DOUBLE)");
try (PreparedStatement insert = connection.prepareStatement(
"INSERT INTO jdbc_sensor VALUES (?, ?, ?)")) {
insert.setLong(1, System.currentTimeMillis() * 1_000_000L);
insert.setString(2, "sensor-1");
insert.setDouble(3, 25.3);
insert.executeUpdate();
}
try (ResultSet result = statement.executeQuery(
"SELECT name, value FROM jdbc_sensor")) {
while (result.next()) {
System.out.printf("%s %.1f%n",
result.getString("NAME"),
result.getDouble("VALUE"));
}
}
}
}
}
```
Use `long` to pass DATETIME values as epoch nanoseconds. If the example table already
exists, omit `CREATE LOG TABLE` or use a different name.
## ROWID from INSERT
After a successful single `INSERT ... VALUES` in Standard Edition, use the standard
JDBC generated keys API to retrieve the inserted row's ROWID.
```java
String sql = "INSERT INTO jdbc_sensor VALUES (?, ?, ?)";
try (PreparedStatement insert = connection.prepareStatement(
sql, Statement.RETURN_GENERATED_KEYS)) {
insert.setLong(1, System.currentTimeMillis() * 1_000_000L);
insert.setString(2, "sensor-2");
insert.setDouble(3, 26.1);
insert.executeUpdate();
try (ResultSet keys = insert.getGeneratedKeys()) {
if (keys.next()) {
java.sql.RowId rowId = keys.getRowId("ROWID");
}
}
}
```
The result has one `ROWID` column and at most one row. If there is no ROWID to return,
the `ResultSet` is empty. Check `DatabaseMetaData.supportsGetGeneratedKeys()` for support.
See [ROWID and INSERT Result IDs](/dbms/reference/sql/rowid/) for differences in batches,
Append, `INSERT ... SELECT`, and UPSERT.
## Check Versions
```java
import java.sql.DatabaseMetaData;
DatabaseMetaData metadata = connection.getMetaData();
System.out.println(metadata.getDriverName());
System.out.println(metadata.getDriverVersion());
System.out.println(metadata.getJDBCMajorVersion()); // 4
System.out.println(metadata.getJDBCMinorVersion()); // 2
```
## Related Documents
| Document | Content |
|------|------|
| [PreparedStatement and Types](./prepared-types/) | Parameter metadata, named binding, SQLType, NULL, and type conversion |
| [ResultSet, Statement, and LOB](./resultset-lob/) | Typed retrieval, streams, LOBs, timeouts, and resource management |
| [Transactions and Connection Pools](./transaction-pooling/) | Standard local transactions, DataSource, and connection pools |
| [DatabaseMetaData](./database-metadata/) | Tables, columns, keys, indexes, and capabilities |
| [Append API](./append-api/) | High-speed ingestion through `MachStatement` |
| [Migration and Troubleshooting](./migration-troubleshooting/) | Migration from earlier drivers, unsupported features, and error handling |
---
title: "11.5.1 PreparedStatement and Types"
url: https://docs.machbase.com/dbms/development-tools-integration/jdbc/prepared-types/
language: en
kind: page
---
# 11.5.1 PreparedStatement and Types
PreparedStatement prepares SQL on the server and executes it repeatedly with different parameter values.
Machbase JDBC supports standard positional parameters and a named-parameter extension.
## ParameterMetaData
Use `PreparedStatement.getParameterMetaData()` to inspect parameter counts and types before execution.
The same API applies to INSERT, UPDATE, and SELECT.
```java
import java.sql.ParameterMetaData;
import java.sql.PreparedStatement;
try (PreparedStatement statement = connection.prepareStatement(
"INSERT INTO sensor_tx " +
"(id, value, created_at) VALUES (?, ?, ?)")) {
ParameterMetaData metadata = statement.getParameterMetaData();
for (int index = 1; index <= metadata.getParameterCount(); index++) {
System.out.printf(
"%d: type=%s precision=%d scale=%d nullable=%d%n",
index,
metadata.getParameterTypeName(index),
metadata.getPrecision(index),
metadata.getScale(index),
metadata.isNullable(index));
}
}
```
Parameter indexes start at 1. An index of 0 or greater than the parameter count raises
`SQLException`. Interpret precision according to the type: for numeric types, it is the JDBC
number of decimal digits, not storage size in bytes. DATETIME maps to
`java.sql.Types.TIMESTAMP`, but its database type name is `DATETIME`.
## Named Bind Parameter
The `:name` placeholder and `MachPreparedStatement.setObject(String, Object)` are Machbase
extensions. When a name occurs more than once, the value is bound to every matching position.
```java
import com.machbase.jdbc.MachPreparedStatement;
try (MachPreparedStatement statement =
(MachPreparedStatement) connection.prepareStatement(
"SELECT id FROM sensor_tx " +
"WHERE id = :id OR parent_id = :id")) {
statement.setObject("id", Integer.valueOf(10));
try (ResultSet result = statement.executeQuery()) {
while (result.next()) {
System.out.println(result.getInt("ID"));
}
}
}
```
- Names may include or omit the leading colon.
- Names are case-sensitive.
- Do not mix named setters and numeric-index setters in one statement.
- An unknown name or mixed named/positional binding raises SQLState `07009`.
- An older server without named binding support raises SQLState `0A000`.
For portability, use the JDBC standard `?` placeholder and numeric-index setters. See
[Named Bind Parameter](/dbms/reference/sql/syntax/named-bind-parameter-syntax/) for the shared syntax.
## Reexecute a Prepared SELECT
You can execute SELECT repeatedly with the same PreparedStatement. Close the previous ResultSet
and bind new values; the next execution returns a new ResultSet.
```java
try (PreparedStatement statement = connection.prepareStatement(
"SELECT id, name FROM sensor_tx WHERE id = ?")) {
ResultSetMetaData metadata = statement.getMetaData();
System.out.println(metadata.getColumnCount());
for (int id = 1; id <= 2; id++) {
statement.setInt(1, id);
try (ResultSet result = statement.executeQuery()) {
while (result.next()) {
System.out.println(result.getString("NAME"));
}
}
}
}
```
Machbase JDBC does not support multiple open results on one Statement. Consume and close each
ResultSet before the next execution. You can retrieve prepare-time `ResultSetMetaData` again
while the Statement remains open.
## JDBC 4.2 SQLType Binding
Use Java 8 `JDBCType` to specify a parameter SQL type.
```java
import java.math.BigDecimal;
import java.sql.JDBCType;
import java.sql.Timestamp;
try (PreparedStatement statement = connection.prepareStatement(
"INSERT INTO sensor_tx " +
"(id, name, value, created_at, payload) " +
"VALUES (?, ?, ?, ?, ?)")) {
statement.setObject(1, Integer.valueOf(1), JDBCType.INTEGER);
statement.setObject(2, "sensor-1", JDBCType.VARCHAR);
statement.setObject(
3, new BigDecimal("12.3400"), JDBCType.DECIMAL, 4);
statement.setObject(
4, Timestamp.valueOf("2026-07-26 10:00:00"),
JDBCType.TIMESTAMP);
statement.setObject(5, new byte[] {1, 2, 3}, JDBCType.BINARY);
statement.executeUpdate();
}
```
An unknown vendor `SQLType` raises `SQLFeatureNotSupportedException`. Use `BigDecimal` for
DECIMAL and NUMERIC; values are bound with the specified scale.
### SQL NULL
`setNull()` or `setObject(index, null, JDBCType)` sends SQL NULL for the target type.
The following types also support typed NULL:
- `REAL`, `BIT`, `TINYINT`, `BOOLEAN`
- `VARBINARY`, `LONGVARBINARY`, `BLOB`, `CLOB`
- `LONGVARCHAR`
NULL for an unsigned parameter is converted to the native NULL value using ParameterMetaData.
The wire NULL sentinel, one greater than the maximum unsigned data value, cannot be stored as
data. Passing that value raises SQLState `22003`.
| Machbase type | Java type | Data range |
|---------------|-----------|-------------|
| `USHORT` | `Integer` | 0–65534 |
| `UINTEGER` | `Long` | 0–4294967294 |
| `ULONG` | `BigInteger` | 0–18446744073709551614 |
Use `setNull()` to insert NULL instead of passing the sentinel directly.
## Boolean
`setBoolean()` and `setObject(index, value, JDBCType.BOOLEAN)` send 1 for `true` and 0 for
`false`. String input accepts only `true` and `false`, case-insensitively. Other values raise
SQLState `22018`.
## IPv4 and IPv6
Use the `MachPreparedStatement` extension setters for IP address columns.
```java
MachPreparedStatement statement =
(MachPreparedStatement) connection.prepareStatement(
"INSERT INTO net_log(ts, src_ip, dst_ip) VALUES (?, ?, ?)");
statement.setLong(1, System.currentTimeMillis() * 1_000_000L);
statement.setIpv4(2, "192.168.1.100");
statement.setIpv6(3, "::1");
statement.executeUpdate();
```
## Nullable Metadata
`ParameterMetaData.isNullable()` returns `parameterNoNulls`, `parameterNullable`, or
`parameterNullableUnknown`. Do not interpret `parameterNullableUnknown` as NOT NULL. See
[Nullable Metadata Support](/dbms/development-tools-integration/sdk-support-scope/#support-scope-sdk-nullable-metadata)
for rules for each SQL statement.
---
title: "11.5.2 ResultSet, Statement, and LOB"
url: https://docs.machbase.com/dbms/development-tools-integration/jdbc/resultset-lob/
language: en
kind: page
---
# 11.5.2 ResultSet, Statement, and LOB
Machbase JDBC ResultSet is a forward-only, read-only cursor. Close Connections, Statements,
and ResultSets with try-with-resources.
```java
statement.getResultSetType(); // ResultSet.TYPE_FORWARD_ONLY
statement.getResultSetConcurrency(); // ResultSet.CONCUR_READ_ONLY
```
Scrollable and updatable ResultSets are not supported. Getter calls before `next()`, after
the last row, or after close, and out-of-range column indexes raise `SQLException`.
## Typed Retrieval
Both `getObject(index, Class)` and `getObject(label, Class)` are supported.
```java
import java.math.BigDecimal;
import java.sql.Timestamp;
try (ResultSet result = statement.executeQuery(
"SELECT id, value, created_at FROM sensor_tx")) {
while (result.next()) {
Integer id = result.getObject("ID", Integer.class);
BigDecimal value =
result.getObject("VALUE", BigDecimal.class);
Timestamp createdAt =
result.getObject("CREATED_AT", Timestamp.class);
}
}
```
| Java type | Typical Machbase type |
|-----------|-------------------------|
| `String` | CHAR, VARCHAR, TEXT, IPV4, IPV6, JSON |
| `Short`, `Integer`, `Long` | Integer types |
| `Float`, `Double` | Floating-point types |
| `BigDecimal` | DECIMAL, NUMERIC |
| `Boolean` | BOOLEAN |
| `Timestamp`, `Date`, `Time` | DATETIME |
| `byte[]` | BINARY |
| `Blob`, `Clob` | BLOB, CLOB |
Object getters return Java `null` for SQL NULL. Primitive getters return 0 or `false`; call
`wasNull()` immediately afterward to check for SQL NULL. Unsupported conversions and a null
target class raise `SQLException`.
In Machbase SQL, the empty string literal `''` is SQL `NULL`. For that result column,
`ResultSetMetaData.isNullable()` returns `columnNullable`, and `getObject()` returns `null`.
The literal `''''` is a single quote character, so it is a non-NULL string.
The default object mappings for unsigned types are:
| Machbase type | `getObject()` return type |
|---------------|-------------------------|
| `USHORT` | `Integer` |
| `UINTEGER` | `Long` |
| `ULONG` | `BigInteger` |
`SHORT` returns an `Integer` object; 32-bit `FLOAT` returns a `Float` object. BOOLEAN strings
accept only `true` and `false`.
## Character and Binary Streams
`setAsciiStream()`, `setBinaryStream()`, and `setCharacterStream()` provide overloads with
`int` length, `long` length, or no length. `setNCharacterStream()` and `setNString()` are
aliases for the VARCHAR path, not a separate NCHAR storage type.
ResultSet supports the following getters by index or column name:
- `getAsciiStream()`, `getBinaryStream()`
- `getCharacterStream()`, `getNCharacterStream()`
- `getNString()`
Input shorter than its declared length, a negative length, or a length greater than
`Integer.MAX_VALUE` raises `SQLException`. Streams currently materialize in client memory;
do not use them for constant-memory streaming of large values.
## BLOB and CLOB
Use standard `Blob` and `Clob` objects to retrieve and bind BLOB/CLOB columns in LOG tables.
```java
import java.io.ByteArrayInputStream;
import java.io.StringReader;
import java.sql.Blob;
import java.sql.Clob;
try (PreparedStatement insert = connection.prepareStatement(
"INSERT INTO event_log (payload, message) VALUES (?, ?)")) {
insert.setBlob(1, new ByteArrayInputStream(payload));
insert.setClob(2, new StringReader(message));
insert.executeUpdate();
}
try (ResultSet result = statement.executeQuery(
"SELECT payload, message FROM event_log")) {
while (result.next()) {
Blob payloadObject = result.getBlob("PAYLOAD");
Clob messageObject = result.getClob("MESSAGE");
byte[] payloadBytes = payloadObject.getBytes(
1, (int) payloadObject.length());
String messageText = messageObject.getSubString(
1, (int) messageObject.length());
payloadObject.free();
messageObject.free();
}
}
```
Create mutable objects with `Connection.createBlob()` and `createClob()`, then use
`setBytes()`, `setString()`, `setBinaryStream()`, `setCharacterStream()`, or `truncate()`.
Call `free()` when finished.
LOB positions are 1-based, as required by JDBC. The entire requested partial-stream range
must fit within the value. A range beyond the end raises SQLState `22003` instead of returning
a truncated result. A negative length or a length that a Java array cannot represent raises
`HY090`. Reusing an object after `free()` raises `SQLException`.
LOBs materialize the entire value in client memory. This is not a streaming LOB implementation
that processes values of hundreds of MiB or more with constant memory.
## ResultSet Metadata
Use `ResultSetMetaData.isNullable()` to inspect nullability of SELECT result columns.
```java
ResultSetMetaData metadata = result.getMetaData();
int nullable = metadata.isNullable(columnIndex);
```
`columnNullableUnknown` does not mean NOT NULL. Check table column constraints with
`DatabaseMetaData.getColumns()` and retrieve PRIMARY KEY information separately with
`DatabaseMetaData.getPrimaryKeys()`.
## Row Counts and Fetch Settings
The JDBC 4.2 large update API returns affected-row counts as `long`.
```java
long count = statement.executeLargeUpdate(
"DELETE FROM sensor_tx WHERE id < 100");
long[] counts = statement.executeLargeBatch();
```
`setLargeMaxRows()` and `getLargeMaxRows()` are also available. `setMaxRows()` or
`setLargeMaxRows()` limits the number of rows retrieved from a ResultSet without changing
the Statement `fetchSize`.
## Statement Lifecycle
- With `closeOnCompletion()`, the Statement closes when its last ResultSet closes.
- Close the previous ResultSet before reexecuting a Statement.
- Commit closes ResultSets, but Statements and PreparedStatements remain reusable.
- Do not call `next()` and getters concurrently from multiple threads on one ResultSet.
## Cancellation and Query Timeout
`Statement.cancel()` cancels the current statement through a separate session. If no statement
is running, it does nothing. PreparedStatement bindings and metadata are preserved.
When `setQueryTimeout(seconds)` expires, it raises `SQLTimeoutException` with SQLState
`HYT00`. After handling the exception, you can reuse the Statement for the next query.
A timeout task from a previous execution does not cancel the next execution.
To run independent queries in parallel, obtain one logical Connection per worker from a
connection pool instead of sharing one Connection. To stop an ongoing fetch, another thread
may call `close()`, `cancel()`, or `Connection.abort()`.
---
title: "11.5.3 Transactions and Connection Pools"
url: https://docs.machbase.com/dbms/development-tools-integration/jdbc/transaction-pooling/
language: en
kind: page
---
# 11.5.3 Transactions and Connection Pools
Machbase JDBC supports standard JDBC local transactions on Standard Edition TRANSACTION tables.
Cluster Edition has no TRANSACTION tables, so the transaction features on this page do not apply.
## Create a TRANSACTION Table
```sql
CREATE TRANSACTION TABLE sensor_tx
(
id INTEGER PRIMARY KEY,
parent_id INTEGER,
name VARCHAR(64),
value DECIMAL(20, 4),
created_at DATETIME,
payload BINARY
);
```
## Commit and Rollback
Use `setAutoCommit(false)`, `commit()`, and `rollback()`. Applications do not need to send
SQL `BEGIN` explicitly.
```java
import java.sql.PreparedStatement;
import java.sql.SQLException;
connection.setAutoCommit(false);
try (PreparedStatement statement = connection.prepareStatement(
"INSERT INTO sensor_tx (id, value) VALUES (?, ?)")) {
statement.setInt(1, 1);
statement.setBigDecimal(2, new BigDecimal("10.5000"));
statement.executeUpdate();
connection.commit();
} catch (SQLException exception) {
try {
connection.rollback();
} catch (SQLException rollbackException) {
exception.addSuppressed(rollbackException);
}
throw exception;
}
```
`setAutoCommit(false)` does not send `BEGIN` immediately. The transaction starts when the
first Statement runs in manual mode. After commit or rollback, auto-commit remains `false`,
and the next Statement starts a new transaction.
- Switching to `setAutoCommit(true)` first commits any active transaction.
- Calling `commit()` or `rollback()` when auto-commit is `true` raises SQLState `25000`.
- Closing a Connection rolls back unfinished transactions.
- Commit and rollback close open ResultSets; Statements remain reusable.
## Isolation Level and Cursors
The supported isolation level is `Connection.TRANSACTION_SERIALIZABLE`. Requesting another
level raises `SQLFeatureNotSupportedException`.
The supported holdability is `ResultSet.CLOSE_CURSORS_AT_COMMIT`. Consume required results
before committing, or execute the query again after commit.
## Behavior by Table Type
| Operation | Behavior in a manual transaction |
|------|--------------------------|
| TRANSACTION table DML/SELECT | Participates in the transaction. |
| LOG/TAG table SELECT | Allowed. |
| First LOG DML before a TRANSACTION change | May be reexecuted with auto-commit through the compatibility path. |
| Standalone TAG DML | Participates in the transaction and can be rolled back. |
| LOG/TAG DML or DDL after a TRANSACTION change | Raises an error. |
LOG DML reexecuted with auto-commit through the compatibility path cannot be rolled back later.
Use TRANSACTION tables for data that requires rollback. Normal commit and rollback across multiple
TRANSACTION tables are supported, but global atomicity is not guaranteed if a failure occurs during
backend commit. Keep critical atomic operations within one TRANSACTION table.
## DataSource
Use `MachDataSource` to supply connection properties to an application server or framework.
```java
import com.machbase.jdbc.MachDataSource;
import java.sql.Connection;
MachDataSource dataSource = new MachDataSource();
dataSource.setUrl("jdbc:machbase://127.0.0.1:5656/machbasedb");
dataSource.setUser("SYS");
dataSource.setPassword(System.getenv("MACHBASE_PASSWORD"));
dataSource.setLoginTimeout(10);
try (Connection connection = dataSource.getConnection()) {
// Execute SQL.
}
```
DataSource supports the URL, user, password, login timeout, log writer, and JDBC `Wrapper` contract.
## ConnectionPoolDataSource
`MachConnectionPoolDataSource` returns logical Connections without exposing physical connections.
```java
import com.machbase.jdbc.MachConnectionPoolDataSource;
import java.sql.Connection;
import javax.sql.PooledConnection;
MachConnectionPoolDataSource source =
new MachConnectionPoolDataSource();
source.setUrl("jdbc:machbase://127.0.0.1:5656/machbasedb");
source.setUser("SYS");
source.setPassword(System.getenv("MACHBASE_PASSWORD"));
PooledConnection pooled = source.getPooledConnection();
try {
try (Connection logical = pooled.getConnection()) {
// Use the logical connection.
}
} finally {
pooled.close();
}
```
Only one logical handle is active per PooledConnection. Closing a logical Connection resets
the following state, then fires `connectionClosed` once:
- Roll back unfinished transactions
- Restore auto-commit
- Restore the initial catalog determined by the URL
- Restore the network timeout
The next connection lease is not issued until calls on the closing logical handle finish.
A closed Connection, Statement, or DatabaseMetaData cannot be reused in the next lease; doing
so raises SQLState `08003`. Statement pooling is not supported.
A fatal connection error in SQLState class `08` discards the physical connection and fires
`connectionErrorOccurred`. Class `23` errors, such as duplicate keys, do not indicate a damaged
connection and do not fire a connection error event.
## HikariCP
```java
import com.zaxxer.hikari.HikariConfig;
import com.zaxxer.hikari.HikariDataSource;
HikariConfig config = new HikariConfig();
config.setJdbcUrl("jdbc:machbase://127.0.0.1:5656/machbasedb");
config.setUsername("SYS");
config.setPassword(System.getenv("MACHBASE_PASSWORD"));
config.setMaximumPoolSize(10);
config.setMinimumIdle(2);
config.setConnectionTimeout(30_000);
config.setIdleTimeout(600_000);
config.setMaxLifetime(1_800_000);
config.addDataSourceProperty("TIMEZONE", "+0900");
try (HikariDataSource dataSource = new HikariDataSource(config);
Connection connection = dataSource.getConnection()) {
// Execute SQL.
}
```
Return logical Connections promptly with try-with-resources. Do not retain returned handles.
## Network timeout
`setNetworkTimeout(executor, milliseconds)` sets the socket read timeout in milliseconds;
`0` means no limit. A negative value, null executor, or executor that rejects tasks raises
`SQLException`.
An actual network timeout raises an exception in SQLState class `08` and invalidates the
physical connection. Obtain a new connection instead of reusing Statements or ResultSets from
the failed connection. I/O failures in active transactions are not retried automatically.
---
title: "11.5.4 DatabaseMetaData"
url: https://docs.machbase.com/dbms/development-tools-integration/jdbc/database-metadata/
language: en
kind: page
---
# 11.5.4 DatabaseMetaData
`Connection.getMetaData()` returns information about the driver, server, schema objects,
and JDBC capabilities. Result columns use JDBC standard names and ordering; read columns
by name rather than numeric position.
```java
import java.sql.DatabaseMetaData;
DatabaseMetaData metadata = connection.getMetaData();
System.out.println(metadata.getDriverName());
System.out.println(metadata.getDriverVersion());
System.out.println(metadata.getDatabaseProductName());
System.out.println(metadata.getDatabaseProductVersion());
```
## Tables and Views
```java
try (ResultSet tables = metadata.getTables(
null, null, "%", new String[] {"TABLE", "VIEW"})) {
while (tables.next()) {
System.out.printf("%s %s%n",
tables.getString("TABLE_NAME"),
tables.getString("TABLE_TYPE"));
}
}
```
`getTables()` distinguishes TABLE from VIEW. See `REMARKS` for the specific Machbase table
type. Applications should use standard column names instead of depending on column positions.
## Columns
```java
try (ResultSet columns = metadata.getColumns(
null, null, "SENSOR_TX", "%")) {
while (columns.next()) {
System.out.printf(
"%s %s size=%d nullable=%s%n",
columns.getString("COLUMN_NAME"),
columns.getString("TYPE_NAME"),
columns.getInt("COLUMN_SIZE"),
columns.getString("IS_NULLABLE"));
}
}
```
`NULLABLE` returns a numeric constant; `IS_NULLABLE` returns `YES`, `NO`, or an empty string.
PRIMARY KEY columns in LOOKUP and VOLATILE tables return `columnNoNulls` and `NO` even
without an explicit NOT NULL clause.
`DECIMAL_DIGITS` and `NUM_PREC_RADIX` are SQL NULL for columns without these numeric
attributes, such as VARCHAR, DATETIME, BINARY, BLOB, and CLOB. Check NULL with `getObject()`
or `wasNull()` instead of relying on a 0 returned by `getInt()`.
## Primary Keys and Indexes
```java
try (ResultSet keys = metadata.getPrimaryKeys(
null, null, "SENSOR_TX")) {
while (keys.next()) {
System.out.printf("%s position=%d%n",
keys.getString("COLUMN_NAME"),
keys.getShort("KEY_SEQ"));
}
}
try (ResultSet indexes = metadata.getIndexInfo(
null, null, "SENSOR_TX", false, false)) {
while (indexes.next()) {
System.out.printf("%s %s%n",
indexes.getString("INDEX_NAME"),
indexes.getString("COLUMN_NAME"));
}
}
```
Do not infer PRIMARY KEY membership from `ResultSetMetaData.isNullable()`.
Use `getPrimaryKeys()` and `getIndexInfo()`.
For a direct column in a SELECT result, the Machbase JDBC extension
`MachResultSetMetaData.isPrimaryKey(column)` reports PRIMARY KEY membership.
```java
import com.machbase.jdbc.MachResultSetMetaData;
try (ResultSet result = statement.executeQuery(
"SELECT ID, ID + 1 AS ID_EXPR FROM SENSOR_TX")) {
MachResultSetMetaData resultMetadata =
(MachResultSetMetaData) result.getMetaData();
System.out.println(resultMetadata.isPrimaryKey(1)); // true or false
System.out.println(resultMetadata.isPrimaryKey(2)); // expression: false
}
```
`getPrimaryKeys()` reads primary keys from the table catalog; `isPrimaryKey()` reads column
metadata for the current SELECT result. An older server or SDK may return `false` for the
result-column primary key flag.
## Schema and Type Information
The following methods return ResultSets in the standard JDBC format:
- `getSchemas()`, `getCatalogs()`, `getTableTypes()`
- `getTypeInfo()`
- `getTables()`, `getColumns()`
- `getPrimaryKeys()`, `getIndexInfo()`
Unsupported optional metadata queries may return an empty ResultSet with standard columns
instead of null or a nonstandard ResultSet. Check the capability methods before using a feature.
```java
if (metadata.supportsSavepoints()) {
// Use savepoints only where supported.
}
```
## catalog
`Connection.getCatalog()` and `setCatalog()` manage the current catalog value exposed by
the driver. Catalog arguments to metadata methods filter requests against this value.
```java
String initialCatalog = connection.getCatalog();
connection.setCatalog(initialCatalog);
```
Returning a logical Connection to a pool restores the initial catalog determined by the URL.
Do not reuse a DatabaseMetaData object from a previous connection lease in the next lease.
## Check Supported Features
Machbase JDBC capability methods reflect actual support. For example, check transaction
isolation levels, ResultSet types, savepoints, generated keys, and multiple open results as follows:
```java
System.out.println(metadata.supportsTransactions());
System.out.println(metadata.supportsTransactionIsolationLevel(
Connection.TRANSACTION_SERIALIZABLE));
System.out.println(metadata.supportsResultSetType(
ResultSet.TYPE_FORWARD_ONLY));
System.out.println(metadata.supportsSavepoints());
System.out.println(metadata.supportsGetGeneratedKeys());
System.out.println(metadata.supportsMultipleOpenResults());
```
A `false` result from `Driver.jdbcCompliant()` is separate from support for individual JDBC
APIs. Applications should check the features they require.
When connected to a Standard Edition server that supports ROWID, `supportsGetGeneratedKeys()`
returns `true`, and `getRowIdLifetime()` returns `ROWID_VALID_OTHER`. On an unsupported server
or Cluster Edition, they return `false` and `ROWID_UNSUPPORTED`, respectively. See
[ROWID and INSERT Result IDs](/dbms/reference/sql/rowid/) for examples.
---
title: "11.5.5 Append API"
url: https://docs.machbase.com/dbms/development-tools-integration/jdbc/append-api/
language: en
kind: page
---
# 11.5.5 Append API
The Machbase Append API ingests many rows in sequence. JDBC exposes it through
`MachStatement` extension methods. This page focuses on LOG ingestion; check the
[SDK Support Matrix](../../sdk-support-scope/#append-table-type-matrix) for other table types.
## API
| Method | Description |
|--------|------|
| `executeAppendOpen(tableName, errorCheckCount)` | Starts an Append session and returns column metadata. |
| `executeAppendOpen(tableName, inputColumns, errorCheckCount)` | Starts an Append session targeting selected columns or ARRAY elements in Machbase DBMS 8.7.0. |
| `executeAppendData(metadata, data)` | Sends one row. |
| `executeAppendDataByTime(metadata, time, data)` | Sends one row with a timestamp in nanoseconds. |
| `executeAppendFlush()` | Synchronizes pending responses. |
| `executeAppendClose()` | Closes the Append session. |
| `executeSetAppendErrorCallback(callback)` | Registers a row-error callback. |
| `getAppendSuccessCount()` | Returns the number of successful rows. |
| `getAppendFailureCount()` | Returns the number of failed rows. |
The public `executeAppendData()` returns `1` on success and throws `SQLException` for an
invalid internal result. Also check final success/failure counts and callback results.
## Ingestion Example
```sql
CREATE LOG TABLE sensor_data (
time DATETIME,
name VARCHAR(40),
value DOUBLE
);
```
```java
import com.machbase.jdbc.MachStatement;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
import java.util.ArrayList;
try (MachStatement statement =
(MachStatement) connection.createStatement()) {
ResultSet appendResult =
statement.executeAppendOpen("sensor_data", 100);
ResultSetMetaData metadata = appendResult.getMetaData();
statement.executeSetAppendErrorCallback(
(errorNumber, errorMessage, rowMessage) ->
System.err.printf(
"Append error [%05d]: %s%n%s%n",
errorNumber, errorMessage, rowMessage));
long baseTime = System.currentTimeMillis() * 1_000_000L;
for (int index = 0; index < 10_000; index++) {
ArrayList