The fundamental problem of communication is reproducing a message from one point to another. — Claude Shannon

The Information Age has revolutionized economy, politics, and culture.

DIS (Digital Information Systems) store, transmit, and transform data with incredible efficiency.

The challenge: Make sense of vast, rapidly growing information.

DIS are digital, interconnected, and concerned with information.

We interact with DIS through apps, webpages, and programs.

(And for that, we need computers and phones)

The purpose of this treatise: Understand DIS systematically.

My hypothesis: these methods yield a 10x speedup and 2-4x quality & value.

My goal: by making DIS simpler, more humans will be empowered to bring their best.

Information = Data + Context. Understanding transforms data into information.

(From now on: data = information)

DIS only do three things:

communicate data
store data
transform data

Speed, accuracy, and cost set digital systems apart from their predecessors.

Digital systems represent data with electrons, not carvings or ink.

The challenge: organize vast amounts of information. Organization is the key.

Most DIS challenges stem from understanding how parts interrelate.

Simple systems are easier to understand than complex ones.

The art of system design is the making of systems that are as simple as possible.

(The limiting factor is complexity, not energy)

When designing or understanding, focus on the data, not The List.

(The painting, not the palette and brushes!)

The List

Programming languages
Programming paradigms
Type systems
Libraries
Communication protocols
Operating systems
System architectures
Performance
Availability
Cost
AI

Why not look at the data?

There's too much data, we cannot look directly at it.
Data is just a detail, it's not important.

(This entire treatise topples these two myths)

We can build on five pillars, each of them a practical concept that removes a major obstacle to looking at the data.

Pillar 1: Single representation of data

Overcomes not being able to look and describe data in unambiguous terms.

Pillar 2: Single dataspace

Overcomes having parts of the system floating around instead of being part of one whole picture.

Pillar 3: Call and response

Overcomes the invisibility of how data is transformed inside a DIS.

Pillar 4: Logic is what happens between call and response

Overcomes doubts about the shape of the solution for a clearly specified problem.

Pillar 5: Interface is call and response

Overcomes separateness between system and user and between data and time.

Digital data is binary. We need to find a better way to represent it than zeroes and ones.

Introducing fourdata

A textual representation for all data.

(Why text? Because it is linear, compact and portable)

Fourdata represents four types of data:

Number:
```
1234
```
Text:
```
Hi
```
List:

1 Eggs 2 Butter 3 Lettuce
Hash:

born 1959 firstName Gustavo lastName Cerati

Data types can be combined and nested

musicians 1 born 1959
            firstName Gustavo
            lastName Cerati
          2 born 1962
            firstName John
            lastName Squire

And can represent data as diverse as an HTTP call

request headers Accept application/json
        host example.com
        method GET
        path /api/books/1234
        type HTTP/1.1
response body author "Edward Said"
              id 1234
              isbn 978-0-394-42814-7
              title Orientalism
         code "200 OK"
         headers Content-Length 83
                 Content-Type application/json

Or the state of a CPU

Accumulator 00110100
ProgramCounter 1100001010101011
StackPointer 11110111
StatusRegister 00100101
X 01100001
Y 11001010

Or a row in a database

books 1 author "Edward Said"
        created 2024-10-23T20:24:15.936Z
        id 1234
        isbn 978-0-394-42814-7
        title Orientalism

Or a simple web page

head title "Welcome to my site"
body 1 h1 "Hello World!"
     2 p 1 class text
         2 "This is my site"

Fourdata can represent any conceivable data directly and without ambiguity, just using text and a few rules.

Every piece of data in our system has a path to it.

A path is a sequence of texts and numbers.

The path to eggs is breakfast 2

breakfast 1 "orange juice"
          2 "eggs"
          3 croissant
          4 yoghurt

Paths are themselves data because they consist of numbers and texts.

Every data point in our system has a path to it.

Paths don't just point to data, they are the data!

Paths make places memorable, associative, even permanent.

To the left, there is context. To the right, detail.

 books 1 author "Edward Said"
<---------------------------->
more context       more detail

Paths are themselves data because they consist of numbers and texts.

Every DIS stores its data in two primary forms: files and databases.

Both can be placed in the dataspace.

For files, put the path to the file as a hash, followed by its content:

C: Users dmr code hello.c "hello world!"

Files can also be represented in binary format

C: Users dmr code hello.c 011010000110010101101100011011000110111100100000011101110110111101110010011011000110010000100001

For databases:

Use a hash to represent database name and table/collection name
A list of hashes for each of the rows/documents

mysql users 1 id fpereiro
            2 id deadmau5

mongo visitors 1 date 2024-11-11T16:42:01.322Z
                 ip 140.28.111.224
               2 date 2024-11-11T16:42:02.299Z
                 ip 220.49.66.236

The dataspace is not where the data is.

The dataspace is the data.

The combination of a call and a response can be used to express any data transformation.

The formula of a call:

@ destination message
= response

@ denotes a call
= denotes its response

A reference to a variable:

value @ widgets
      = 100
widgets 100

A function call:

@ + 1 10
    2 10
= 20

A database query:

users @ "Main Database" "select * from users"
      = 1 id 1
          username deadmau5
        2 id 2
          username faxingberlin

An HTTP call:

orientalism @ http headers Accept application/json
                   host example.com
                   method GET
                   path /api/user/1234
            = body id 1234
                   username deadmau5

An assembler instruction:

@ mov eax 0x1A3F
= 00100001

Call and response represent the dynamic nature of data while still being data.

Logic is how a call creates a response.

(Logic is intentional transformation of data).

The five elements of logic:

Reference: destination of a call.
Sequence: calls made by a call.
Conditional: choice between sequences.
Loop: conditional repetition of a sequence.
Error: special type of conditional response.

The first three are essential, the last two are nice to have.

A reference is the destination of a call.

References are links between parts of the dataspace.

A reference can point to a mere value:

@ widgets
= 100

And it also can point to a call:

@ + 1 10
    2 10
= 20

Resolving a reference is finding what part of the dataspace it refers to.

Here is one way to do it:

From the place where the call is made, we go one level up (left) and try to find it. If it's not there, we repeat the process of going one more level up to find it.
If we have gone all the way to the left and we can't find it, we obtain an empty text.

A sequence is a list of calls.

The concept of sequence works at any level of abstraction.

Function, flow, procedure, operation, definition, all express the same: a list of calls.

A good analogy for a sequence is a recipe:

"make cake" @ : 1 @ mix . @ chocolate 100
                        . @ flour 500
                        . @ butter 300
                2 @ bake degrees 200
                         ingredients @ 1

The colon (:) freezes the sequence so that it can be expanded only when it is called.

When we call a sequence, we can see its expansion:

@ "make cake" 8
: 1 @ mix . @ chocolate @ * . @ people
                              = 8
                            . 25
                        = 200
            = "200 grams of chocolate!"
          . @ flour @ * . @ people
                          = 8
                        . 125
                    = 1000
            = "A kilo of flour!"
          . @ butter @ * . @ people
                           = 8
                         . 75
                     = 600
            = "600 grams of butter!"
    = "A mix of 200g of chocolate, a kilo of flour and 600g of butter!"
  2 @ bake degrees 200
           ingredients @ 1
    = "chocolate cake for eight!"
= "chocolate cake for eight!"

Note the colon (:) contains the expansion of the calls.

Conditionals let you choose between sequences based on a condition:

@ if cond 1
     do @ "make cake" 8
        = "chocolate cake for eight!"
= "chocolate cake for eight!"

The simplest conditional only has one sequence, which will only be expanded if the condition is true.

An example with two sequences:

dinner @ if cond @ party
                 = 0
            do @ "make cake" 8
            else @ "make healthy dish" 2
                 = "Beetroot soup!"
       = "Beetroot soup!"
party 0

Loops are conditional repetition of sequences:

@ loop data . 1
            . 2
            . 4
       do @ "plus ten"
= 1 11
  2 12
  3 14

This simple type of loop will likely be like at least 50% of the loops in your logic.

Loops can be used as filters:

evens @ loop data . 1
                  . 2
                  . 3
                  . 4
             filter @ : value @ even @ value
      = . 2
        . 4

Or as accumulators:

"gimme ten" @ loop acc @ +
                   data . 1
                        . 2
                        . 3
                        . 4
             = 10

Recursion can be understood as loops with depth:

flatten : v @ if cond @ type type list
                             value @ v
                 do loop data @ v
                         do @ flatten
                 else @ add to "flat list"
                            value @ v

If calls can represent transformation at any level, we can go beyond declarative vs imperative.

The "what" is the interface, the "how" is its logic. Every call is declarative on the outside, imperative on the inside.

Everyday notion of interface: something made for humans, with graphics.

Formal notion of interface: a boundary between two parts of the system.

A more general way to look at it: an interface is the combination of a call and its response.

logic "what transforms a call into a response"
interface 1 call
          2 response

Implications:

No intrinsic boundary between user and system, both are one.
No intrinsic "internal" or "external" areas. A call has an external area (interface) and an internal area (logic).
User calls and system calls are the same.

Calls are reactive.

When something changes (destination, message, logic), the response is updated.

When a part of the system changes, the system updates itself to stay in sync. This is the true meaning of reactivity.