Bulk CSV import

graphbuff import builds a new persistent database from CSV files in the neo4j-admin database import format. It writes store records directly: no Cypher, no redo log, no transaction size cap. On 100,000 nodes and 1,000,000 relationships (with properties) it takes about 9 s, where LOAD CSV with CREATE takes about 140 s.

graphbuff import --data data --database social \
    --nodes Person=people_header.csv,people1.csv,people2.csv \
    --relationships FOLLOWS=follows.csv \
    --index 'Person(id)'

people_header.csv   id:ID(Person),name,born:int,tags:string[],:LABEL
follows.csv         :START_ID(Person),:END_ID(Person),since:date

• Headers: name:ID(space) or :ID (the import id; with a name it is also stored as a property), :LABEL, :START_ID(space), :END_ID(space), :TYPE, :IGNORE, and properties name:type where the type is int, long, float, double, boolean, string, point, date, localtime, time, localdatetime, datetime or duration, or an array such as string[]. A plain name is a string. The first record of a group's first file is the header; the group's other files have none.
• Options: --delimiter, --array-delimiter (also separates labels, default ;), --id-type string|integer, --skip-bad-relationships, --skip-duplicate-nodes, --index 'Label(a, b)' or --index '[:TYPE](a)' (built after loading, from sorted keys), --cache-mb, --no-fsync. graphbuff import --help lists them all.
• Target: --data DIR adds the database to that data directory's catalog (default name: the default database). An existing database is replaced only if it is empty, or with --overwrite. A failed import leaves it as it was. The server must not be running on DIR (the catalog lock keeps it out). --into STORE_DIR writes a bare persistent store into a new directory instead.
• From Rust: DatabaseManager::import_database, or PersistentGraph::bulk_loader plus import_csv for a bare store. BulkLoader can also be filled directly (create_node, create_relationship, create_index, finish).

Nothing is durable until the import finishes. While it runs, the store directory holds an IMPORTING marker, and opening a directory that still has one fails: delete it and import again. Import ids are kept in memory (one map per id space), as are the label index and property index keys until they are written. Constraints are not created by the import; add them with CREATE CONSTRAINT afterwards (with an index from --index, it seeks instead of building its own).

$ docker run --rm -p 127.0.0.1:7475:7475 -p 127.0.0.1:7687:7687 ghcr.io/janvencl/graphbuff --demo movies