Skip to content

README.md

Latest commit

 

History

History
249 lines (191 loc) · 8.8 KB

README.md

File metadata and controls

249 lines (191 loc) · 8.8 KB
Raw

English | Chinese

TsFile Tools Manual

Introduction

Development

Prerequisites

To build the Java version of TsFile Tools, you must have the following dependencies installed:

  1. Java >= 1.8 (1.8, 11 to 17 are verified. Make sure the environment variable is set).
  2. Maven >= 3.6 (if you are compiling TsFile from source).

Build with Maven

mvn clean package -P with-java -DskipTests

Install to local machine

mvn install -P with-java -DskipTests

Hybrid CSV Import

Combine one main time-series CSV (with a real time column) and multiple supplement CSVs (same TAG/FIELD columns, no time column) into a single TsFile. Supplement rows receive synthetic timestamps 1, 2, …, N per file; each file is isolated with a virtual TAG batch_id (one ChunkGroup per file per business TAG combination).

Example config (hybrid.conf):

output_tsfile=combined.tsfile
shared_schema=main.schema
main_csv=timeseries.csv
main_batch_id=main
batch_id_tag=batch_id
validate_uniform_tags=true
supplement_sort_by_variance=true
supplement_csv=experiment_1.csv
supplement_batch_id=experiment_1
supplement_csv=experiment_2.csv
supplement_batch_id=experiment_2

Run:

java -jar tsfile-tools.jar --hybrid_config hybrid.conf

Supplement CSV headers must list all business TAG and FIELD columns from shared_schema, excluding the time column (e.g. Region,DeviceId,Temperature,Pressure).

For each supplement CSV separately (supplement_sort_by_variance=true by default):

  1. Compute variance of each FIELD column within that CSV only.
  2. Order columns by variance descending (higher variance = higher sort priority).
  3. Sort rows in that CSV ascending (multi-key comparator).
  4. Write one ChunkGroup per CSV; timestamps are consecutive inside the group (startId, startId+1, …).
  5. The next supplement CSV continues ids from maxId + 1 (file1: 1..n1, file2: n1+1..n1+n2, …).

Programmatic API: HybridCsvTsFileAssembler.execute(HybridImportConfig).

Schema Definition

Parameters

Parameter Description Required Default
table_name Table name Yes
time_precision Time precision (ms / us / ns / s) No ms
has_header Whether CSV contains a header (true / false). Ignored for Parquet / Arrow. No true
separator CSV delimiter (, / tab / ;). Ignored for Parquet / Arrow. No ,
null_format String value treated as null in CSV. Ignored for Parquet / Arrow (native null). No
tag_columns Tag columns (device identifiers / primary key). Supports virtual columns with DEFAULT value. No
time_column Time column name Yes
source_columns Column definitions mapping to source file columns Yes

Backward compatibility: id_columns and csv_columns are still accepted as aliases for tag_columns and source_columns.

Column Concepts

  • time_column: Exactly one per table. Written as time column with type TIMESTAMP in TsFile.
  • tag_columns: Device identifiers (composite primary key), 0 or more. Supports virtual columns not present in the source file via DEFAULT keyword.
  • source_columns: Maps every column in the source file by position (CSV) or by name (Parquet / Arrow). Use SKIP to ignore a column.
  • FIELD (derived, not configured): All columns in source_columns that are not time_column, not in tag_columns, and not SKIP. These are the measurement columns whose values change over time.

Schema Example

CSV file content:

Region,FactoryNumber,DeviceNumber,Model,MaintenanceCycle,Time,Temperature,Emission
hebei,1001,1,10,1,1,80.0,1000.0
hebei,1001,1,10,1,4,80.0,1000.0
hebei,1002,7,5,2,1,90.0,1200.0

Schema file (import.schema):

table_name=root.db1
time_precision=ms
has_header=true
separator=,
null_format=\N

tag_columns
Group DEFAULT Datang
Region
FactoryNumber
DeviceNumber

time_column=Time

source_columns
Region TEXT,
FactoryNumber TEXT,
DeviceNumber TEXT,
SKIP,
SKIP,
Time INT64,
Temperature FLOAT,
Emission DOUBLE,

In this example:

  • Group is a virtual tag column (not in CSV) with default value Datang
  • Region, FactoryNumber, DeviceNumber are tag columns read from CSV
  • Model and MaintenanceCycle are skipped via SKIP
  • Temperature and Emission are automatically derived as FIELD columns

For Parquet / Arrow in schema mode, source_columns matches by column name instead of position. Named SKIP is also supported:

source_columns
Time INT64,
unused_col SKIP,
Temperature FLOAT,
Emission DOUBLE,

CLI Parameters

Parameter Description Required Default
-s, --source Input file or directory Yes
-t, --target Output directory Yes
--schema Schema file path. Omit for auto mode. No
--fail_dir Directory for failed source files No failed
--format Source format: csv / parquet / arrow. Auto-detected by file extension if omitted. No auto-detect
--table_name Table name override (auto mode) No derived from filename
--time_precision Time precision override (auto mode): ms / us / ns / s No ms
--separator CSV delimiter (auto mode): , / tab / ; No ,
-b, --block_size CSV chunk size (e.g. 256M, 1G) No 256M
-tn, --thread_num Thread count for parallel processing No 8

Modes

Schema Mode

Provide a --schema file to explicitly define column mapping, types, tags, and time column.

# CSV
csv2tsfile.sh --source ./data/csv --target ./output --fail_dir ./failed --schema ./schema/import.schema
csv2tsfile.bat --source .\data\csv --target .\output --fail_dir .\failed --schema .\schema\import.schema

# Parquet
parquet2tsfile.sh --source ./data/parquet --target ./output --fail_dir ./failed --schema ./schema/import.schema
parquet2tsfile.bat --source .\data\parquet --target .\output --fail_dir .\failed --schema .\schema\import.schema

# Arrow
arrow2tsfile.sh --source ./data/arrow --target ./output --fail_dir ./failed --schema ./schema/import.schema
arrow2tsfile.bat --source .\data\arrow --target .\output --fail_dir .\failed --schema .\schema\import.schema

Auto Mode

Omit --schema to automatically infer column types and detect the time column.

Auto mode rules:

  • Time column: must be named exactly time or TIME (case-sensitive, strict match)
  • All other columns become FIELD (no tag inference)
  • CSV type inference uses a 100-row sampling window. Promotion rules: INT64 and DOUBLE promote to DOUBLE; any other mixed pair (including BOOLEAN with numeric) promotes to STRING.
  • Parquet / Arrow use native schema types directly
  • Default table name: derived from source filename (e.g. sensor.csv → table sensor)
  • Default null tokens (CSV only): empty cell and \N

Auto mode example:

CSV file (sensor.csv):

time,temperature,humidity,status
1000,25.5,60.0,true
2000,26.1,55.3,false
3000,27.0,58.1,true

Auto mode infers:

table name:  sensor        (from filename)
time column: time
fields:      temperature DOUBLE, humidity DOUBLE, status BOOLEAN
tags:        (none)

Commands:

# CSV
csv2tsfile.sh --source ./data/csv --target ./output --fail_dir ./failed
csv2tsfile.bat --source .\data\csv --target .\output --fail_dir .\failed

# CSV with options
csv2tsfile.sh --source ./data/csv --target ./output --table_name my_table --separator tab --time_precision us

# Parquet
parquet2tsfile.sh --source ./data/parquet --target ./output --fail_dir ./failed
parquet2tsfile.bat --source .\data\parquet --target .\output --fail_dir .\failed

# Arrow (.arrow / .ipc / .feather)
arrow2tsfile.sh --source ./data/arrow --target ./output --fail_dir ./failed
arrow2tsfile.bat --source .\data\arrow --target .\output --fail_dir .\failed

Output File Naming

  • Single batch: {source_basename}.tsfile
  • Multiple batches: {source_basename}_1.tsfile, {source_basename}_2.tsfile, ...
  • Table name and output filename are independent — table name comes from schema or --table_name, filename comes from source file.