Generate

foundry generate produces test data with Alchemy, YndigoBlue's test-data generation API. You describe the tables you want and the fields that populate their columns in a JSON payload, and Alchemy returns data that follows it, deterministically when the payload sets a seed.

foundry generate dataset [options]   # one or more related tables, as a .csvx package
foundry generate table [options]     # a single table, as a bare .csv file

A dataset pairs with foundry import schema, which builds the schema the package describes and loads every table into it. A single table pairs with foundry import table, which loads it into a table that already exists.

To check a payload and see what it would cost before generating anything, add --estimate.

Options

Both subcommands take the same options:

Option	Short	Description
`--endpoint <URL>`	`-e`	Base URL of the Alchemy API, e.g. `https://alchemy.yndigoblue.com`. Falls back to the `ALCHEMY_ENDPOINT` environment variable
`--api-key <KEY>`	`-k`	Alchemy API key, sent as `X-Api-Key`. Falls back to the `ALCHEMY_API_KEY` environment variable
`--payload <FILE>`	`-p`	Path to the JSON request payload (required)
`--out <FILE>`	`-o`	Path of the file to write (required, unless `--estimate`)
`--estimate`	`-E`	Check and price the payload without generating anything or being billed. See Estimating Cost

Tip

Set ALCHEMY_API_KEY in your environment rather than passing --api-key, so the key never ends up in a run script or your shell history.

Dataset

foundry generate dataset sends the payload to Alchemy's POST /v1/generate-dataset and writes the .csvx package it returns: a ZIP holding a manifest.json, a Velocity schema file describing every generated table (columns, primary keys, and the foreign keys between tables), and one CSV per table.

foundry generate dataset \
  --endpoint https://alchemy.yndigoblue.com \
  --payload dataset.json \
  --out dataset.csvx

foundry import schema --datasource connection.json --file dataset.csvx

Table

foundry generate table sends the payload to Alchemy's POST /v1/generate-table and writes the one table it describes as a bare CSV file, header row included, with no manifest or schema.

foundry generate table \
  --endpoint https://alchemy.yndigoblue.com \
  --payload customer.json \
  --out customer.csv

foundry import table --datasource connection.json --schema dbo --table customer --file customer.csv

Writing a Payload

Every Foundry project's schemas/ folder includes a JSON Schema for each payload shape, so an editor such as VS Code offers completion for every field type and its settings, and flags a mistake as you type rather than when the request is refused:

Schema	Payload for
`schemas/alchemy-dataset.schema.json`	`foundry generate dataset`
`schemas/alchemy-table.schema.json`	`foundry generate table`

Point the payload at its schema with $schema:

{
  "$schema": "schemas/alchemy-dataset.schema.json",
  "seed": 42,
  "schemaName": "shop",
  "tables": [
    {
      "name": "customer",
      "rowCount": 100,
      "primaryKey": ["id"],
      "fields": [
        { "type": "Number", "mode": "Sequential", "start": 1,
          "map": { "value": { "column": "id", "dataType": "Integer" } } },
        { "type": "Person", "countries": [{ "country": "us" }],
          "map": {
            "firstName": { "column": "first_name", "dataType": "VarChar" },
            "lastName": { "column": "last_name", "dataType": "VarChar" } } }
      ]
    },
    {
      "name": "orders",
      "rowCount": 500,
      "primaryKey": ["order_id"],
      "fields": [
        { "type": "Number", "mode": "Sequential", "start": 1000,
          "map": { "value": { "column": "order_id", "dataType": "Long" } } },
        { "type": "Reference", "targetTable": "customer", "targetColumn": "id",
          "map": { "value": { "column": "customer_id", "dataType": "Integer" } } }
      ]
    }
  ]
}

A table payload is the same shape as one ordinary entry in tables, with seed, schemaVersion, and csvFormat alongside it at the top level. A link table ("type": "Link") copies its rows from another table in the same request, so it's only available in a dataset payload:

{
  "$schema": "schemas/alchemy-table.schema.json",
  "seed": 42,
  "name": "customer",
  "rowCount": 100,
  "fields": [
    { "type": "Number", "mode": "Sequential", "start": 1,
      "map": { "value": { "column": "id", "dataType": "Integer" } } }
  ]
}

Foundry sends the payload file to Alchemy exactly as it is. Alchemy accepts the top-level $schema and ignores it, but rejects any other property it doesn't recognize, so a misspelled setting fails the request rather than being silently dropped.

See Alchemy's own documentation for every field type and its settings.

Note

A .csvx package records the csvFormat its tables were written in, so foundry import schema reads it with no extra options. A bare CSV from foundry generate table carries no such record: if its payload sets csvFormat, pass the matching options to foundry import table (--delimiter, --encoding, and so on).

Output

On success Foundry reports how many bytes of data were generated and, when Alchemy reports one, what the call cost:

Dataset written to dataset.csvx (.csvx)
368 bytes generated, cost 0.0001 USD

When Alchemy refuses a request, nothing is written and Foundry prints the reason Alchemy gave, including where in the payload the problem is:

Error: Alchemy returned 400 Bad Request: Request body could not be read. Unknown property 'bogus'
at $.fields[0].bogus. ...

Estimating Cost

Add --estimate to either subcommand to send the payload to Alchemy's matching estimate endpoint instead. It runs the same checks a generation request runs before it generates anything, then reports what the request would cost and how much of each per-call limit it uses. No data is generated, nothing is written, and the call isn't billed, so --out isn't needed:

foundry generate dataset --endpoint https://alchemy.yndigoblue.com --payload dataset.json --estimate

dataset.json passed Alchemy's checks.
Estimated cost: 0.0042 USD
Balance: 12.50 USD (covers the estimate)
┌────────────────────────┬───────────┬─────────────┬────────┐
│ Limit                  │ Requested │     Maximum │   Used │
├────────────────────────┼───────────┼─────────────┼────────┤
│ Rows                   │       110 │     100,000 │  0.1 % │
│ Columns (widest table) │         8 │          25 │ 32.0 % │
│ Data bytes (estimated) │    12,380 │ 268,435,456 │  0.0 % │
│ Calculation steps      │         0 │  50,000,000 │  0.0 % │
└────────────────────────┴───────────┴─────────────┴────────┘

A payload that fails a check, or asks for more than a limit allows, is refused with the same error a generation request would get. --estimate also fails when your balance doesn't cover the estimate, so a run script stops before it reaches the generation step. Without an API key, which only a Development instance of Alchemy accepts, the payload is still checked but not priced.

Note

An estimate isn't a guaranteed price, and it doesn't reserve anything. A generation request is charged at the price in effect when it runs. Some checks need the generated data, such as a primaryKey column turning out to contain a repeated value, so a payload that passes an estimate can still be refused when it's generated.

In a Run Script

Both subcommands are available as generate-dataset and generate-table steps in a run script, with "estimate": true for an estimate, so a project can check its payload, generate its test data, and load it in one foundry run:

{
  "$schema": "schemas/run-script.schema.json",
  "datasource": "connection.json",
  "steps": [
    {
      "name": "Check the payload and the balance",
      "generate-dataset": {
        "endpoint": "https://alchemy.yndigoblue.com",
        "payload": "dataset.json",
        "estimate": true
      }
    },
    {
      "name": "Generate test data",
      "generate-dataset": {
        "endpoint": "https://alchemy.yndigoblue.com",
        "payload": "dataset.json",
        "out": "dataset.csvx"
      }
    },
    {
      "name": "Load it",
      "import-schema": { "file": "dataset.csvx", "drop": true }
    }
  ]
}

With ALCHEMY_API_KEY set in the environment, the script needs no key of its own.

Table of Contents

Generate

Options

Tip

Dataset

Table

Writing a Payload

Note

Output

Estimating Cost

Note

In a Run Script