Getting Started with Shards

Introduction to Shards

Shards is a data flow programming language where data moves through pipelines of operations called "shards". Unlike traditional programming languages, Shards focuses on data transformation through pipes (|) rather than function calls.

Data Flow Programming

In traditional programming, you might write:

result = process(validate(parse(input)))

In Shards, data flows left to right through pipes:

input | Parse | Validate | Process = result

This flow-based approach makes data transformations clear and intuitive:

; Multiple operations in a flow
"5" | FromJson | Math.Add(3) | Log    ; "5" -> 5 -> 8 -> print 8

; Multiple operations on same input using blocks
input | {
  Math.Add(1) | Log         ; First operation
  Math.Multiply(2) | Log    ; Second operation on original input
}

Core Concepts

1. Wires

Wires are the basic building blocks in Shards. They define a pipeline of operations:

; Basic wire definition
@wire(example {
  "Hello" | Log              ; Data flows through pipes
  5 | Math.Add(3) = result   ; Store results in variables
})

2. Variables & Assignment

Shards has three types of assignment:

; Immutable assignment (=)
5 = constant                  ; Can't be changed
"data" = immutable-string

; Mutable declaration (>=)
0 >= counter                  ; Can be updated
"" >= message

; Update mutable (>)
counter | Math.Add(1) > counter    ; Increment counter
"new text" > message               ; Update message

3. Basic Types

; Numbers
42 = integer
3.14 = float
0xFF = hex-number

; Strings
"Hello World" = text

; Vectors
@f3(1.0 2.0 3.0) = float-vec3    ; 3D float vector
@i2(640 480) = int-vec2          ; 2D integer vector

; Sequences (lists)
[1 2 3 4] = numbers              ; Number sequence
["a" "b" "c"] = strings          ; String sequence

; Tables (dictionaries)
{
  name: "Alice"
  age: 30
  scores: [85 92 78]
} = user-data

4. Control Flow

; Conditionals
value | If(IsMore(10)
  {"Greater than 10" | Log}
  {"Less or equal to 10" | Log}
)

; When condition
input | When(IsEmpty {
  "Input is empty" | Error
})

; ForEach loop
[1 2 3] | ForEach({
  Math.Multiply(2) | Log    ; Prints: 2, 4, 6
})

5. Error Handling

; Try-catch pattern
Maybe({
  "bad-json" | FromJson     ; Try this
} {
  "Parse failed" | Log      ; Handle error
})

; Validation with error
input | When(Not(IsString) {
  "Expected string input" | Error
})

Using Shards in edge talk

In edge talk, we use Shards to create tools that process data and interact with external services. A typical tool has this structure:

; Inner wire that does the work
@wire(tool-inner {
  ; Get input parameter
  {Take("input") | ExpectString = input}
  
  ; Process the input
  input | Process | Transform
} Pure: true)    ; Pure ensures clean state

; Main wire that runs our tool
@wire(tool-name {
  Do(tool-inner)
})

; Tool definition for edge talk
{
  definition: {
    name: "my_tool"
    description: "What this tool does"
    parameters: {
      type: "object"
      properties: {
        input: {
          type: "string"
          description: "The input to process"
        }
      }
      required: ["input"]
    }
  }
  
  use: tool-name
}

With these fundamentals in mind, let's look at how to create practical tools with Shards.

Key Concepts

Data Flow Programming

In traditional programming, you might write:

result = process(validate(parse(input)))

In Shards, data flows left to right through pipes:

input | Parse | Validate | Process = result

Tool Structure

Edge talk tools typically have three parts:

; Inner wire that does the actual work
@wire(tool-inner {
  ; Process inputs and return result
} Pure: true)

; Main wire that executes our tool
@wire(tool-name {
  Do(tool-i)
})

; Tool definition for edge talk
{
  definition: {
    name: "tool_name"
    description: "What the tool does"
    parameters: {
      type: "object"
      properties: {
        param1: {
          type: "string"
          description: "Parameter description"
        }
      }
      required: ["param1"]
    }
  }
  
  use: tool-name
}

Your First Tool

Let's create a simple tool that processes and validates user input:

; Process user data
@wire(process-user-inner {
  ; Extract and validate input
  {Take("name") | ExpectString = name}
  {Take("email") | ExpectString = email}
  
  ; Validate email format
  email | When(Not(Match("^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}$")) {
    "Invalid email format" | Error
  })
  
  ; Format output
  {
    name: name
    email: email
    timestamp: Time.Now
    validated: true
  } | ToJson
} Pure: true)

; Main wire
@wire(process-user {
  Do(process-user-inner)
})

; Tool definition
{
  definition: {
    name: "process_user"
    description: "Process and validate user information"
    parameters: {
      type: "object"
      properties: {
        name: {
          type: "string"
          description: "User's full name"
        }
        email: {
          type: "string"
          description: "User's email address"
        }
      }
      required: ["name" "email"]
    }
  }
  
  use: process-user
}

Data Types & Validation

Input Validation

; Common validation patterns
{Take("input") | ExpectString = str-input}
{Take("count") | ExpectInt = int-input}
{Take("data") | ExpectTable = table-input}
{Take("items") | ExpectSeq = seq-input}

; Optional inputs with defaults
{Take("limit") | Default("10") | ExpectString = limit}
{Take("offset") | Default("0") | ExpectString = offset}

String Operations

; String formatting with mixed types
["User " name " created at " timestamp] | String.Format = message

; String manipulation
input | Match("pattern") = matched
input | Split(" ") = words
["a" "b" "c"] | String.Join = joined

Tables & JSON

; Create table
{
  id: "123"
  name: "Test"
  values: [1 2 3]
} = data-table

; Convert to/from JSON
data-table | ToJson = json-str
json-str | FromJson | ExpectTable = parsed-table

Error Handling

; Validation and errors
input | When(IsEmpty {
  "Input cannot be empty" | Error
})

; Try-catch pattern
Maybe({
  json-str | FromJson | Process
} {
  "Failed to process JSON" | Error
})

; Conditional processing
value | If(IsMore(10)
  {Process.Large(value)}
  {Process.Small(value)}
)

HTTP Tools

@wire(fetch-data-inner {
  ; Setup headers
  {
    "Authorization": (["Bearer " token] | String.Join)
    "Content-Type": "application/json"
  } = headers
  
  ; Make request
  {
    limit: "10"
    offset: "0"
  } | Http.Get("https://api.example.com/data" Headers: headers)
  
  ; Process response
  FromJson | ExpectTable
  Take("items") | ExpectSeq
  ForEach({
    ExpectTable
    Process.Item
  })
} Pure: true)

Best Practices

• Always validate inputs with Expect* shards
• Use Pure: true for tool wires to ensure clean state
• Provide clear error messages
• Document parameters thoroughly
• Break complex operations into smaller wires
• Use consistent naming conventions