Method ImportData

Namespace: YndigoBlue.Velocity.Engine

Assembly: YndigoBlue.Velocity.dll

ImportData(Table, string, ImportConfig, bool)

Imports data from a CSV file into a database table.

public void ImportData(Table table, string dataFilePath, ImportConfig importConfig = null, bool streamImport = false)

Parameters

table Table: The Table to import data into.
dataFilePath string: The path to the CSV file containing the data to import.
importConfig ImportConfig: Optional ImportConfig specifying delimiters, mappings, and other import settings. Uses defaults if null.
streamImport bool: When true the file is read and sent to the database in chunks of ImportBatchSize rows instead of being loaded into memory in full first. Defaults to false.

Examples

using (var m = new Manager(conn))
{
    var schema = m.LoadSchema("app");
    var usersTable = schema["users"];

    // Simple import with default settings
    m.ImportData(usersTable, "users.csv");

    // Import with custom configuration
    var config = new ImportConfig(usersTable);
    config.FieldDelimiter = '|';
    config.UseColumnHeaders = true;
    config.Mappings.AddMapping("UserName", "name", DataType.VarChar);
    config.Mappings.AddMapping("EmailAddress", "email", DataType.VarChar);

    m.ImportData(usersTable, "users_pipe_delimited.csv", config);

    // Stream a large file rather than loading it all into memory
    m.ImportData(usersTable, "users_10m_rows.csv", config, streamImport: true);
}

Remarks

By default the whole CSV file is parsed into memory before any of it is sent to the database, so peak memory use scales with the size of the file. Setting streamImport to true reads the file in chunks of ImportBatchSize rows and sends each chunk as it is read, which keeps peak memory flat regardless of file size. Prefer it for large files.

Streaming changes neither what ends up in the table nor the meaning of ImportTransaction — a failure part-way through leaves the table exactly as a failure during a whole-file import would. The trade-off is that the import holds its transaction, and any locks that implies, for as long as the file takes to read.

Exceptions

DbException: Thrown when a database error occurs.
InvalidImportMappingsException: Thrown when column mappings are invalid.
ValueSetForAutoGeneratedColumnException: Thrown when attempting to import into an auto-increment column.