
I don't write summaries of the year. They are prone to be too personal to be interesting to others. But 2021 has been different to me.

The Chinese New Year has just passed. Looking back at the passing year, I realized an overarching theme that runs through various events. It's about entering new areas and having new perspectives. It makes me rethink many things that I have been taking for granted for a long time.

It's been a year of exploration.

## Logseq

At the beginning of the year, I started working at [Logseq](https://logseq.com), an open-source note-taking app with bidirectional linking. It's a new experience for me in many ways. I will list them in the order of ascending significance.

### First Full-time Remote Job

The obvious advantage is that I get to work with amazing people all around the world. For example, Logseq mostly uses the Clojure programming language, which I have never used before. But since the founder [Tienshen](https://twitter.com/tiensonqin) and other team members are excellent Clojure programmers, I picked it up by learning from them pretty fast.

As a programmer, my duties are mostly the same. But the flow of work has changed a lot.

I make my own schedule and work alone most of the time. This optimizes efficiency and flexibility. On the other hand, sometimes it makes me feel lonely and directionless, missing the social interactions of an office.

The communication cost is much higher. Zoom calls can be tiring and lose many subtleties of an in-person talk. Instant-messaging apps can be nerve-wracking. Ideally, this will be mitigated by asynchronous communication. Basecamp has a [good guide](https://basecamp.com/guides/how-we-communicate) on this topic.

There are many other things that I've learned working remotely. To make it short, I'd say it comes with its unique pros and cons and requires a different mindset and work culture to be successful.

### First Time Working on an Open-Source Project as a Day Job

Getting to work a on fairly large (and [popular](https://github.com/logseq/logseq)) open-source project is the most attractive thing when applying for the job, from which the most important thing I learned is that it's all about the community.

At Logseq, the whole development process is centered around the community. We take feature requests and bug reports from the users. After each milestone, we base the next one on user feedback. To encourage contribution to the project, we spend much effort writing docs and tutorials. It's a remarkable lesson that you can learn so much by talking and listening to the community carefully.

### First Entrance to the Tools for Thought Territory

Before joining Logseq I've only lightly used traditional note-taking apps. Upon first trying apps like Logseq and Roam Research, I'm overwhelmed by all the new features. Bidirectional links? Networked thoughts? Digital gardens?

Not for long, it all click for me. I see why these features are useful and better than traditional ones. More importantly, I understand that computers as a tool for thought have so many unfulfilled potentials.

As I build features for Logseq, I start to explore the tools for thought landscape. There are many ongoing efforts and some early products, yet we are still at a very early stage of using computers to empower the human mind. This sensation is best captured in Andy Matuschak's [How can we develop transformative tools for thought](https://numinous.productions/ttft/).

## Local-First Software

As a web developer, most of the apps that I have built so far can be categorized as cloud-based web apps - browser GUIs connected to data and business logic on the server.

On the contrary, Logseq is local-first. Most of the user data is stored on the device and can be synced optionally. This is good for performance, privacy, and data ownership while being much more complicated to build than their cloud counterparts.

I learned a lot from the article [Local-first software](https://www.inkandswitch.com/local-first/). The article not only illustrates the concept of local-first software but also recommends [CRDT](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type) as a foundational technology. The team behind it even build some interesting libraries and prototypes.

There is one caveat. The proposed methods are suitable for building peer-to-peer apps instead of client-server apps. And I'm more interested in the latter. After some research, I find a few helpful resources and libraries. Similar to tools for thought development, this area is at a rather young age.

I managed to build a [prototype](https://todo-light.vercel.app) and to write an [article](https://zjy.cloud/posts/collaborative-web-apps) about it, which [gained some love](https://news.ycombinator.com/item?id=28209736) from Hacker News.

Currently, I'm building a side project that helps people learn better, combining what I've learned from my probe into tools for thought and local-first software. I have no idea what will come out of it. Let's keep on exploring.

## APL

I'm a big podcast listener. One late night in September, I played one episode from the CoRecursive podcast named [From Competitive Programming to APL](https://corecursive.com/065-competitive-coding-with-conor-hoekstra/). It's an interview with the programmer [Conor Hoekstra](https://twitter.com/code_report).

In the beginning, it sounds like a usual programmer interview. How the protagonist got interested in computers. How he got into the field. What is he doing right now. Etc.

But about halfway through the episode, Conor starts to talk about how he coincidentally discovered an ancient programming language called APL and fell in love with it. He mentions that the APL is weird at first look because it consists of hieroglyphic symbols like Chinese characters, which can be combined to write extremely terse code. And the only data structure APL has is the multi-dimensional array.

I was instantly hooked, partly because I'm Chinese and partly because I feel there's something both alien and profound about this lost art. So I delve in, learning the language, reading its history, and watching many videos of people talking about it. It's a fascinating journey.

When some people encourage fellow programmers to learn Lisp, they often quote from <Sidenote content={<span>Interestingly, Alan J. Perlis once wrote an article called <a target="_blank" href="https://www.jsoftware.com/papers/perlis77.htm">In Praise of APL</a> which eloquently captures many of APL's merits.</span>}>Alan J. Perlis</Sidenote>,

> A language that doesn't affect the way you think about programming, is not worth knowing.

APL fits this saying well. More than Lisp, it has changed how I think about programming and computer science. What's more, it reminds me to always be humble and open-minded.


The Year of Exploration


In my blog post <a href="/posts/collaborative-web-apps">A Simple Way to Build Collaborative Web Apps</a>, I built a demo using the awesome <a href="https://replicache.dev">Replicache</a> in-browser database. At the end of that post, I mentioned that Replicache at that time has two modes: memory and persistent, controlled by the `useMemstore` parameter. The persistent mode is not fast enough to support quick user interactions, such as drag and drop in <a href="http://todo-light.vercel.app">Todo Light</a>. As a result, I chose to use the memory mode for faster interactions, giving up on persistence.

The good news is that the Replicache team just released <a href="https://github.com/rocicorp/replicache/releases/tag/v9.0.0-beta.0">a beta version</a> which implements a unified storage model. To quote the release note, "it's as fast than the old memory mode, but also persists every few seconds to storage so that data can be reused across instances.". You can read the detailed release note <a href="https://replicache.notion.site/9-0-0-beta-0-3cb9e838342b4eeab95908b78252aa40">here</a>.

I've upgraded <a href="http://todo-light.vercel.app">Todo Light</a> to the new version and it works smoothly. The upgrade process is effortless. Kudos to the Replicache team for keep making this already amazing software better.


Todo Light Improvement


Recently I've been thinking about how to build a collaborative web app in 2021.

By _collaborative web app_ I mean apps with _desktop-like interactions_ and _realtime collaborations_, such as Notion, Discord, Figma, etc.

I want to find an approach suitable for _small teams_, _simple_ but not necessarily [easy](https://www.infoq.com/presentations/Simple-Made-Easy/), _fast_ by default, and _scalable_ when it grows.

Thus I started a journey to find out by creating a demo todo app, exploring the <Sidenote content="I welcome all kinds of third-party tools and services, as long as they don't bring in hard-to-replace vendor lock-ins.">tools</Sidenote> and methods along the way.

Our todo app has these features:

- users can create, edit, delete and reorder todos, which are stored on the server for _persistency_
- users can _cooperatively_ edit the same list of todos, and changes are automatically synced between different clients in _realtime_
- all the operations should be as _fast_ as possible for all the users around the world

The _fast_ part is mainly concerned with _latency_, because most apps won't have much _throughput_ to deal with in the beginning. More specifically, we want the changes made by one client to be delivered to other clients in a portion of the speed of light - less than 100ms in the same geographical region and several hundred ms across the continent. The app is named Todo Light for this reason.

You can play with the <Sidenote content="The result failed to meet our goal for speed, depending on your location in the world. More on this later.">end product</Sidenote> [here](https://todo-light.vercel.app/). The [client](https://github.com/thezjy/todo-light-browser-app) and [server](https://github.com/thezjy/todo-light-api) code are both hosted on Github.

<Image
  src="/todo-light.png"
  alt="todo light app screenshot"
  width={1432}
  height={838}
  layout="responsive"
/>

A random list id is created when the app starts. You can share the URL with the _list_id_ parameter to collaboratively edit the same list of todos with others (or yourself in another browser tab).

Todo Light is simple enough as a demo but nonetheless embodies some essence of the SaaS apps mentioned above, albeit in a much-simplified manner.

Let's get started building it.

## Client

We begin from the client because it contains the core of our app.

The user interface part is easy, which we use React to build. Other reactive UI frameworks like Svelte and Vue should work as well. I choose React because of familiarity.

The client-only version of the app is straightforward to write:

```jsx
import React, { useState } from 'react'

export default function TodoApp() {
  const [todos, setTodos] = useState([])
  const [content, setContent] = useState('')

  return (
    <div>
      <form
        onSubmit={(e) => {
          e.preventDefault()
          if (content.length > 0) {
            setTodos((todos) => {
              return [...todos, { content, completed: false }]
            })
            setContent('')
          }
        }}
      >
        <input
          autoFocus
          value={content}
          onChange={(e) => setContent(e.target.value)}
        />
      </form>
      <ul>
        {todos.map((todo, index) => {
          return (
            <li key={index}>
              <span>{todo.content}</span>

              <div>
                <label>
                  <input
                    type="checkbox"
                    checked={todo.completed}
                    onChange={(e) => {
                      setTodos((todos) => {
                        return todos.map((todo, i) => {
                          if (i === index) {
                            return {
                              ...todo,
                              completed: e.target.checked,
                            }
                          } else {
                            return todo
                          }
                        })
                      })
                    }}
                  />
                </label>

                <button
                  onClick={() => {
                    setTodos((todos) => {
                      return todos.filter((todo, i) => {
                        return index !== i
                      })
                    })
                  }}
                >
                  x
                </button>
              </div>
            </li>
          )
        })}
      </ul>
    </div>
  )
}
```

With less than 100 lines of code, the resulting app already looks and works like the end product, except that its state is volatile. Refresh the browser, and you'd lose all your todo items!

We use React's _state_ to store data, which is okay for the input value because it is temporary by nature, but not quite right for the todos. The todos need to be:

1. <Sidenote content="This pattern is sometimes called Optimistic UI">updated in a local browser cache for maximal speed</Sidenote>
2. synced to the server for persistency
3. delivered to other clients in correct order and state.

Nowadays, there is a plethora of frontend state management libraries to choose from: Redux, MobX, Recoil, GraphQL clients like Apollo and Relay, etc. Sadly none of them works in our use case. What we need is a distributed system with realtime syncing and conflict resolution baked in. Although there are [good writings](https://www.figma.com/blog/how-figmas-multiplayer-technology-works) on this subject, distributed systems are still too hard to implement correctly for a one-person team. I'd like to bring in some help.

After some search, a promising option shows up - [Replicache](https://replicache.dev/), of which the homepage says:

> Replicache makes it easy to add realtime collaboration, lag-free UI, and offline support to web apps. It works with any backend stack.

Sounds too good to be true (spoiler: it's mostly true). How does Replicache achieve these bold claims? Its doc site has [a whole page](https://doc.replicache.dev/how-it-works) to explain how it works. To save your time, I will summarize roughly here.

Replicache implements a persistent store in the browser, using IndexedDB. You can mutate the store locally and subscribe to part of the store in your UI. When data changes, subscriptions re-fire, and the UI refreshes.

You need to provide two backend endpoints for Replicache to talk to: _replicache-pull_ and _replicache-push_. _replicache-pull_ sends back a subset of your database for the current client. _replicache-push_ updates the database from local mutations. After applying a mutation on the server, you
send a <Sidenote content={<span> As the Replicache doc says, managing your own WebSocket backend has a very high operational cost. We use <a target="_blank" href="https://ably.com">Ably</a> here.</span>}>WebSocket</Sidenote> message hinting to affected clients to pull again.

That's all you need to do. Relicache orchestrates the whole process to make sure the state is consistent while being synced in realtime.

We will dive into the backend integration in the next section of this article. For now, let's rewrite the state-related code utilizing Replicache:

```jsx
// Only relevant part are shown

import { Replicache } from 'replicache'
import { useSubscribe } from 'replicache-react'
import { nanoid } from 'nanoid'

const rep = new Replicache({
  // other replicache options
  mutators: {
    async createTodo(tx, { id, completed, content, order }) {
      await tx.put(`todo/${id}`, {
        completed,
        content,
        order,
        id,
      })
    },
    async updateTodoCompleted(tx, { id, completed }) {
      const key = `todo/${id}`
      const todo = await tx.get(key)
      todo.completed = completed

      await tx.put(`todo/${id}`, todo)
    },
    async deleteTodo(tx, { id }) {
      await tx.del(`todo/${id}`)
    },
  },
})

export default function TodoApp() {
  const todos =
    useSubscribe(rep, async (tx) => {
      return await tx.scan({ prefix: 'todo/' }).entries().toArray()
    }) ?? []

  const onSubmit = (e) => {
    e.preventDefault()
    if (content.length > 0) {
      rep.mutate.createTodo({
        id: nanoid(),
        content,
        completed: false,
      })

      setContent('')
    }
  }

  const onChangeCompleted = (e) => {
    rep.mutate.updateTodoCompleted({
      id: todo.id,
      completed: e.target.checked,
    })
  }

  const onDelete = (_e) => {
    rep.mutate.deleteTodo({ id: todo.id })
  }

  // render
}
```

We replace React's in-memory state with Replicache's persistent store. The app should work as before, except your carefully written todo items won't disappear when the browser tab closes.

Notice the _mutators_ we register when initializing Replicache. They are the main APIs we use to interact with Replicache's store. When they are executed on the client, the corresponding mutations will be sent to the _replicache-push_ endpoint by Replicache.

With the help of Replicache, you can think about your client state as a giant [hashtable](https://roamresearch.com/#/app/aboodman/page/zfq46G7Ta). You can read from it and write to it as you like, and Replicache would dutifully keep the state in sync among the server and all the clients.

## Server

Now let's move on to the server.

The plan is clear: we will implement the two endpoints needed by Replicache, using some backend language (we use NodeJS in this case) and some database. The only requirement by Replicache is that the database must support a certain kind of [transaction](https://doc.replicache.dev/launch-checklist#all-endpoints).

Before we set out to write the code, we need to think about the architecture. Remember the third feature of Todo Light? It should be as fast as possible for all users around the world.

Since we have implemented Optimistic UI on the client, most operations are already speedy (zero latency). For changes to be synced from one client to others quickly, we still need to achieve low latency for the requests to the server. Hopefully, the latency should be under 100ms for the collaboration to feel _realtime_.

We can only achieve that by globally deploying the server and the database. If we don't and only deployed to one region, the latency for a user in another continent will be several hundred milliseconds high no matter what we do. It's the speed of light, period.

Globally deploying a stateless server should be easy. At least that's what I initially thought. Turns out I was wrong. In 2021, most <Sidenote content="Mostly I'm referring to PaaS like Heroku and Google App Engine. FaaS (function as a service) is much easier to deploy globally but comes with its own gotchas.">cloud</Sidenote> providers still only allow you to deploy your server to a single region. You need to go many extra steps to have a global setup.

Luckily I find [Fly.io](https://fly.io/), a cloud service that helps you "deploy app servers close to your users", which is excatly what we need. It comes with an excellent command-line tool and a smooth "push to deploy" deployment flow. Scaling out to multiple regions (in our case, Hong Kong and Los Angeles) takes only a few keystrokes. Even better, they offer a pretty generous free tier.

The only question left is which database we should use. Globally distributed databases with strong consistency is a huge and complicated area that has been tackled by big companies in recent years.

Inspired by Google's [Spanner](https://research.google/pubs/pub39966/), many open source solutions come out. One of the most polished competitors is [CockroachDB](https://www.cockroachlabs.com/). Luckily, they offer a managed service with a 30-day trial.

Although I managed to build a version of Todo Light using CockroachDB, the end product in this article is based on a much simpler [Postgres setup](https://fly.io/blog/globally-distributed-postgres/) with distributed read replicas. Dealing with a global database brings in much complexity that is not essential to the subject matter of this article, which will wait for another piece.

We need two tables, one for todos and one for replicache clients.

<Image
  src="/db-schema.png"
  alt="database schema"
  width={1260}
  height={852}
  layout="responsive"
/>

Replicache needs to track the _last_mutation_id_ of different clients to coordinate all mutations, whether confirmed or pending. The _deleted_ column is used for soft deletes. The _version_ column is used to compute change for Replicache pulls, which we will explain later.

The _replicache-push_ endpoint receives arguments from the local mutators. Let's persist them to the database. We also need to increment the _lastMutationID_ in the same transaction, as mandated.

```javascript
router.post('/replicache-push', async (req, res) => {
  const { list_id: listID } = req.query
  const push = req.body

  try {
    // db is a typical object than represents a database connection

    await db.tx(async (t) => {
      let lastMutationID = await getLastMutationID(t, push.clientID)

      for (const mutation of push.mutations) {
        const expectedMutationID = lastMutationID + 1

        if (mutation.id < expectedMutationID) {
          console.log(
            `Mutation ${mutation.id} has already been processed - skipping`,
          )
          continue
        }

        if (mutation.id > expectedMutationID) {
          console.warn(`Mutation ${mutation.id} is from the future - aborting`)
          break
        }

        // these mutations are automatically sent by Replicache when we execute their counterparts on the client
        switch (mutation.name) {
          case 'createTodo':
            await createTodo(t, mutation.args, listID)
            break
          case 'updateTodoCompleted':
            await updateTodoCompleted(t, mutation.args)
            break
          case 'updateTodoOrder':
            await updateTodoOrder(t, mutation.args)
            break
          case 'deleteTodo':
            await deleteTodo(t, mutation.args)
            break
          default:
            throw new Error(`Unknown mutation: ${mutation.name}`)
        }
        lastMutationID = expectedMutationID
      }

      // after successful mutations we use Ably to notify the clients
      const channel = ably.channels.get(`todos-of-${listID}`)
      channel.publish('change', {})

      await t.none(
        'UPDATE replicache_clients SET last_mutation_id = $1 WHERE id = $2',
        [lastMutationID, push.clientID],
      )

      res.send('{}')
    })
  } catch (e) {
    console.error(e)

    res.status(500).send(e.toString())
  }
})

async function getLastMutationID(t, clientID) {
  const clientRow = await t.oneOrNone(
    'SELECT last_mutation_id FROM replicache_clients WHERE id = $1',
    clientID,
  )

  if (clientRow) {
    return parseInt(clientRow.last_mutation_id)
  }

  await t.none(
    'INSERT INTO replicache_clients (id, last_mutation_id) VALUES ($1, 0)',
    clientID,
  )

  return 0
}

async function createTodo(t, { id, completed, content, order }, listID) {
  await t.none(
    `INSERT INTO todos (
     id, completed, content, ord, list_id) values
     ($1, $2, $3, $4, $5)`,
    [id, completed, content, order, listID],
  )
}

async function updateTodoCompleted(t, { id, completed }) {
  await t.none(
    `UPDATE todos
     SET completed = $2, version = gen_random_uuid()
     WHERE id = $1
     `,
    [id, completed],
  )
}

// other similar SQL CRUD functions are omitted
```

The _replicache-pull_ endpoint requires more effort. The general plan is, in every request to _replicache-pull_ we compute a _diff of state_ and an arbitrary _cookie_ (not to be confused with HTTP cookie) to send back to the client. The cookie will be attached to the subsequent request to compute the diff. Rinse and repeat.

How to compute the diff may be the most challenging part of integrating Replicache. The team provides several helpful [strategies](https://dusty-lancer-e6c.notion.site/Diff-Strategies-3cccf424123042ed8796cf0773cacf4e). We will use the most recommend one: _the row version strategy_.

```javascript
router.post('/replicache-pull', async (req, res) => {
  const pull = req.body
  const { list_id: listID } = req.query

  try {
    await db.tx(async (t) => {
      const lastMutationID = parseInt(
        (
          await t.oneOrNone(
            'select last_mutation_id from replicache_clients where id = $1',
            pull.clientID,
          )
        )?.last_mutation_id ?? '0',
      )

      const todosByList = await t.manyOrNone(
        'select id, completed, content, ord, deleted, version from todos where list_id = $1',
        listID,
      )

      // patch is an array of mutations that will be applied to the client
      const patch = []

      const cookie = {}

      // For initial call we will just clear the client store.
      if (pull.cookie == null) {
        patch.push({ op: 'clear' })
      }

      todosByList.forEach(
        ({ id, completed, content, ord, version, deleted }) => {
          // The cookie is a map from row id to row version.
          // As the todos count grows, it might become too big to be efficiently exchanged.
          // By then, we can compute a hash as a cookie and store the actual cookie on the server.
          cookie[id] = version

          const key = `todo/${id}`

          if (pull.cookie == null || pull.cookie[id] !== version) {
            if (deleted) {
              patch.push({
                op: 'del',
                key,
              })
            } else {
              // addtions and updates are all represented as the 'put' op
              patch.push({
                op: 'put',
                key,
                value: {
                  id,
                  completed,
                  content,
                  order: ord,
                },
              })
            }
          }
        },
      )

      res.json({ lastMutationID, cookie, patch })
      res.end()
    })
  } catch (e) {
    res.status(500).send(e.toString())
  }
})
```

Because _version_ is a random UUID generated by Postgres's _gen_random_uuid_ function, we can use it to efficiently calculate whether a todo item has been updated or not.

That's all for the server code, and we've come to the end of our journey. With the help of many great tools, we've successfully built a fast, collaborative todo app. More importantly, we've worked out a reasonably simple approach to building similar web apps. As the user base and feature set grow, this approach shall scale well in both performance and complexity.

## Bonus - Implement Reordering with Fractional Indexing

You may notice that we use the type _text_ for the _ord_ column in the database schema, which seems better suited for a number type. The reason is we are using a technique called [Fractional Indexing](https://github.com/rocicorp/fractional-indexing) to implement reordering. Check the [source code](https://github.com/rocicorp/fractional-indexing) of Todo Light or try to implement it by yourself. It should be an interesting practice.

At the time of the writing, one shortcoming of Replicache is that its local transactions are not fast enough to enable heavy interactions such as drag and drop. To prevent lagging, we turned on the ` useMemstore: true` option to disable offline support. Hopefully, this will be fixed soon.

<hr />

#### Update

- [2022-06-30] It works again!
- [2022-05-25] The Replicache shortcoming mentioned at the end of this article has been [fixed](https://github.com/rocicorp/replicache/releases/tag/v9.0.0-beta.0).
- [2022-05-25] The demo is not working right now. I will setup fly.io and deploy it soon.

<hr />
