Skip to content
Effect V4 Docs

Ndjson.ts

Ndjson.ts overview

Section titled “Ndjson.ts overview”

Encodes and decodes newline-delimited JSON streams in Effect channels.

NDJSON stores one complete JSON value on each line. This module has helpers for byte streams, string streams, and schema-checked records, so streaming code can read or write one JSON record at a time.

Since v4.0.0

Exports Grouped by Category

Section titled “Exports Grouped by Category”

combinators

Section titled “combinators”

duplex

Section titled “duplex”

Wraps a bidirectional byte channel with NDJSON encoding and decoding.

Details

Outgoing values are written as UTF-8 NDJSON bytes, and incoming bytes are parsed as NDJSON values.

Signature

declare const duplex: {
  (options?: {
    readonly ignoreEmptyLines?: boolean | undefined
  }): <R, IE, OE, OutDone, InDone>(
    self: Channel.Channel<
      Arr.NonEmptyReadonlyArray<Uint8Array>,
      OE,
      OutDone,
      Arr.NonEmptyReadonlyArray<Uint8Array>,
      IE | NdjsonError,
      InDone,
      R
    >
  ) => Channel.Channel<
    Arr.NonEmptyReadonlyArray<unknown>,
    NdjsonError | OE,
    OutDone,
    Arr.NonEmptyReadonlyArray<unknown>,
    IE,
    InDone,
    R
  >
  <R, IE, OE, OutDone, InDone>(
    self: Channel.Channel<
      Arr.NonEmptyReadonlyArray<Uint8Array>,
      OE,
      OutDone,
      Arr.NonEmptyReadonlyArray<Uint8Array>,
      IE | NdjsonError,
      InDone,
      R
    >,
    options?: { readonly ignoreEmptyLines?: boolean | undefined }
  ): Channel.Channel<
    Arr.NonEmptyReadonlyArray<unknown>,
    NdjsonError | OE,
    OutDone,
    Arr.NonEmptyReadonlyArray<unknown>,
    IE,
    InDone,
    R
  >
}

Source

Since v4.0.0

duplexSchema

Section titled “duplexSchema”

Wraps a bidirectional byte channel with schema-aware NDJSON encoding and decoding.

Details

Values sent to the wrapped channel are encoded with inputSchema; bytes received from it are parsed as NDJSON and decoded with outputSchema.

Signature

declare const duplexSchema: {
  <In extends Schema.Constraint, Out extends Schema.Constraint>(options: {
    readonly inputSchema: In
    readonly outputSchema: Out
    readonly ignoreEmptyLines?: boolean | undefined
  }): <OutErr, OutDone, InErr, InDone, R>(
    self: Channel.Channel<
      Arr.NonEmptyReadonlyArray<Uint8Array>,
      OutErr,
      OutDone,
      Arr.NonEmptyReadonlyArray<Uint8Array>,
      NdjsonError | Schema.SchemaError | InErr,
      InDone,
      R
    >
  ) => Channel.Channel<
    Arr.NonEmptyReadonlyArray<Out["Type"]>,
    NdjsonError | Schema.SchemaError | OutErr,
    OutDone,
    Arr.NonEmptyReadonlyArray<In["Type"]>,
    InErr,
    InDone,
    R | In["EncodingServices"] | Out["DecodingServices"]
  >
  <Out extends Schema.Constraint, In extends Schema.Constraint, OutErr, OutDone, InErr, InDone, R>(
    self: Channel.Channel<
      Arr.NonEmptyReadonlyArray<Uint8Array>,
      OutErr,
      OutDone,
      Arr.NonEmptyReadonlyArray<Uint8Array>,
      NdjsonError | Schema.SchemaError | InErr,
      InDone,
      R
    >,
    options: { readonly inputSchema: In; readonly outputSchema: Out; readonly ignoreEmptyLines?: boolean | undefined }
  ): Channel.Channel<
    Arr.NonEmptyReadonlyArray<Out["Type"]>,
    NdjsonError | Schema.SchemaError | OutErr,
    OutDone,
    Arr.NonEmptyReadonlyArray<In["Type"]>,
    InErr,
    InDone,
    R | In["EncodingServices"] | Out["DecodingServices"]
  >
}

Source

Since v4.0.0

duplexSchemaString

Section titled “duplexSchemaString”

Wraps a bidirectional string channel with schema-aware NDJSON encoding and decoding.

Details

Values sent to the wrapped channel are encoded with inputSchema; strings received from it are parsed as NDJSON and decoded with outputSchema.

Signature

declare const duplexSchemaString: {
  <In extends Schema.Constraint, Out extends Schema.Constraint>(options: {
    readonly inputSchema: In
    readonly outputSchema: Out
    readonly ignoreEmptyLines?: boolean | undefined
  }): <OutErr, OutDone, InErr, InDone, R>(
    self: Channel.Channel<
      Arr.NonEmptyReadonlyArray<string>,
      OutErr,
      OutDone,
      Arr.NonEmptyReadonlyArray<string>,
      NdjsonError | Schema.SchemaError | InErr,
      InDone,
      R
    >
  ) => Channel.Channel<
    Arr.NonEmptyReadonlyArray<Out["Type"]>,
    NdjsonError | Schema.SchemaError | OutErr,
    OutDone,
    Arr.NonEmptyReadonlyArray<In["Type"]>,
    InErr,
    InDone,
    R | In["EncodingServices"] | Out["DecodingServices"]
  >
  <Out extends Schema.Constraint, In extends Schema.Constraint, OutErr, OutDone, InErr, InDone, R>(
    self: Channel.Channel<
      Arr.NonEmptyReadonlyArray<string>,
      OutErr,
      OutDone,
      Arr.NonEmptyReadonlyArray<string>,
      NdjsonError | Schema.SchemaError | InErr,
      InDone,
      R
    >,
    options: { readonly inputSchema: In; readonly outputSchema: Out; readonly ignoreEmptyLines?: boolean | undefined }
  ): Channel.Channel<
    Arr.NonEmptyReadonlyArray<Out["Type"]>,
    NdjsonError | Schema.SchemaError | OutErr,
    OutDone,
    Arr.NonEmptyReadonlyArray<In["Type"]>,
    InErr,
    InDone,
    R | In["EncodingServices"] | Out["DecodingServices"]
  >
}

Source

Since v4.0.0

duplexString

Section titled “duplexString”

Wraps a bidirectional string channel with NDJSON encoding and decoding.

Details

Outgoing values are written as NDJSON strings, and incoming strings are parsed as NDJSON values.

Signature

declare const duplexString: {
  (options?: {
    readonly ignoreEmptyLines?: boolean | undefined
  }): <R, IE, OE, OutDone, InDone>(
    self: Channel.Channel<
      Arr.NonEmptyReadonlyArray<string>,
      OE,
      OutDone,
      Arr.NonEmptyReadonlyArray<string>,
      IE | NdjsonError,
      InDone,
      R
    >
  ) => Channel.Channel<
    Arr.NonEmptyReadonlyArray<unknown>,
    NdjsonError | OE,
    OutDone,
    Arr.NonEmptyReadonlyArray<unknown>,
    IE,
    InDone,
    R
  >
  <R, IE, OE, OutDone, InDone>(
    self: Channel.Channel<
      Arr.NonEmptyReadonlyArray<string>,
      OE,
      OutDone,
      Arr.NonEmptyReadonlyArray<string>,
      IE | NdjsonError,
      InDone,
      R
    >,
    options?: { readonly ignoreEmptyLines?: boolean | undefined }
  ): Channel.Channel<
    Arr.NonEmptyReadonlyArray<unknown>,
    NdjsonError | OE,
    OutDone,
    Arr.NonEmptyReadonlyArray<unknown>,
    IE,
    InDone,
    R
  >
}

Source

Since v4.0.0

constructors

Section titled “constructors”

decode

Section titled “decode”

Creates a channel that decodes UTF-8 byte chunks and parses them as NDJSON.

Details

Lines may span input chunks, and ignoreEmptyLines controls whether blank lines are skipped before JSON parsing.

Signature

declare const decode: <IE = never, Done = unknown>(options?: {
  readonly ignoreEmptyLines?: boolean | undefined
}) => Channel.Channel<
  Arr.NonEmptyReadonlyArray<unknown>,
  IE | NdjsonError,
  Done,
  Arr.NonEmptyReadonlyArray<Uint8Array>,
  IE,
  Done
>

Source

Since v4.0.0

decodeSchema

Section titled “decodeSchema”

Creates an NDJSON byte decoder channel for values of a schema.

Details

The channel decodes UTF-8 bytes, parses each NDJSON line, and then decodes each parsed value with the schema.

Signature

declare const decodeSchema: <S extends Schema.Constraint>(
  schema: S
) => <IE = never, Done = unknown>(options?: {
  readonly ignoreEmptyLines?: boolean | undefined
}) => Channel.Channel<
  Arr.NonEmptyReadonlyArray<S["Type"]>,
  Schema.SchemaError | NdjsonError | IE,
  Done,
  Arr.NonEmptyReadonlyArray<Uint8Array>,
  IE,
  Done,
  S["DecodingServices"]
>

Source

Since v4.0.0

decodeSchemaString

Section titled “decodeSchemaString”

Creates an NDJSON string decoder channel for values of a schema.

Details

The channel parses each line as JSON and then decodes each parsed value with the schema.

Signature

declare const decodeSchemaString: <S extends Schema.Constraint>(
  schema: S
) => <IE = never, Done = unknown>(options?: {
  readonly ignoreEmptyLines?: boolean | undefined
}) => Channel.Channel<
  Arr.NonEmptyReadonlyArray<S["Type"]>,
  Schema.SchemaError | NdjsonError | IE,
  Done,
  Arr.NonEmptyReadonlyArray<string>,
  IE,
  Done,
  S["DecodingServices"]
>

Source

Since v4.0.0

decodeString

Section titled “decodeString”

Creates a channel that parses NDJSON string chunks into values.

When to use

Use when NDJSON input arrives as string chunks and each complete line should be parsed into a JSON value.

Details

Lines may span input chunks.

Gotchas

Set ignoreEmptyLines to skip blank lines before calling JSON.parse; otherwise blank lines are parsed and fail as invalid JSON.

Signature

declare const decodeString: <IE = never, Done = unknown>(options?: {
  readonly ignoreEmptyLines?: boolean | undefined
}) => Channel.Channel<
  Arr.NonEmptyReadonlyArray<unknown>,
  IE | NdjsonError,
  Done,
  Arr.NonEmptyReadonlyArray<string>,
  IE,
  Done
>

Source

Since v4.0.0

encode

Section titled “encode”

Creates a channel that encodes chunks of values as UTF-8 NDJSON bytes.

Signature

declare const encode: <IE = never, Done = unknown>() => Channel.Channel<
  Arr.NonEmptyReadonlyArray<Uint8Array>,
  IE | NdjsonError,
  Done,
  Arr.NonEmptyReadonlyArray<unknown>,
  IE,
  Done
>

Source

Since v4.0.0

encodeSchema

Section titled “encodeSchema”

Creates an NDJSON byte encoder channel for values of a schema.

Details

Values are first encoded with the schema and then written as UTF-8 newline-delimited JSON.

Signature

declare const encodeSchema: <S extends Schema.Constraint>(
  schema: S
) => <IE = never, Done = unknown>() => Channel.Channel<
  Arr.NonEmptyReadonlyArray<Uint8Array>,
  NdjsonError | Schema.SchemaError | IE,
  Done,
  Arr.NonEmptyReadonlyArray<S["Type"]>,
  IE,
  Done,
  S["EncodingServices"]
>

Source

Since v4.0.0

encodeSchemaString

Section titled “encodeSchemaString”

Creates an NDJSON string encoder channel for values of a schema.

Details

Values are first encoded with the schema and then written as newline-delimited JSON strings.

Signature

declare const encodeSchemaString: <S extends Schema.Constraint>(
  schema: S
) => <IE = never, Done = unknown>() => Channel.Channel<
  Arr.NonEmptyReadonlyArray<string>,
  NdjsonError | Schema.SchemaError | IE,
  Done,
  Arr.NonEmptyReadonlyArray<S["Type"]>,
  IE,
  Done,
  S["EncodingServices"]
>

Source

Since v4.0.0

encodeString

Section titled “encodeString”

Creates a channel that encodes chunks of values as NDJSON strings.

Details

Each input item is JSON.stringify-encoded, separated by newlines, and the output chunk ends with a trailing newline.

Signature

declare const encodeString: <IE = never, Done = unknown>() => Channel.Channel<
  Arr.NonEmptyReadonlyArray<string>,
  IE | NdjsonError,
  Done,
  Arr.NonEmptyReadonlyArray<unknown>,
  IE,
  Done
>

Source

Since v4.0.0

errors

Section titled “errors”

NdjsonError (class)

Section titled “NdjsonError (class)”

Error raised when NDJSON encoding or decoding fails.

Details

The kind field identifies whether the failure happened while packing or unpacking, and cause preserves the original error.

Signature

declare class NdjsonError

Source

Since v4.0.0

[NdjsonErrorTypeId] (property)

Section titled “[NdjsonErrorTypeId] (property)”

Marks this value as an NDJSON encoding or decoding error for runtime guards.

Signature

readonly [NdjsonErrorTypeId]: "~effect/encoding/Ndjson/NdjsonError"

Source

Since v4.0.0