Blockchain mail (b-mail)

March 2021 Draft - still thinking about this.

I keep coming back to this idea of an email-like system based on smart contracts.

The core proposal is that there should be economic incentive to only communicate valuable information. Sending an e-mail should cost "something".

Why doesn't this exist now? Possibly because the obvious way to do it involves a centralised authority which charges you (eg a post-office), and that isn't appealing for multiple reasons. But actually we could implement a system like this quite easily with blockchain (a mutually trusted mechanism of transferring wealth) and smart-contracts (a way to write decentralised software).

As a convenient bonus, encryption is first class.

The core challenge is to determine the pricing mechanism, and, ideally, the network itself would determine the pricing mechanism (eg automatically penalising spammers). A secondary question is whether we need a way of establishing some kind of trust between parties.

So how would this work?

The basic mechanism is straightforward:

  • To send a message to someone, you use their public key to encrypt and wrap the message (headers+body) into a token and publish it on the blockchain.
  • A smart contract takes up your message, determines a suitable messaging fee, and transfers the fee from your account to your recipient's account, pushing the message to the recipient at the same time.

    • There's some encryption magic to ensure the message is only readable once the smart contract has processed it.
  • The recipient decrypts your message with their private key and reads it.
  • After that, and depending on the messaging fee algorithm, or network rules, anything more complex could be done.

Pricing structures

The cost to send a message could be fixed or determined dynamically. For example, it could based on

  • The size of the message.
  • Some other property of the message content (eg "information density")
  • The number of times you've messaged that person before
  • The number of messages you send per T (rolling time window), which could discourage sending many messages
  • The "correspondence factor" of the two participants (eg ratio of to/from each party, possibly with a time-factor). This could reduce the cost for friends who talk often.
  • Could the contract involve a "contacts list"? ie the message is free if both people agree for it to be free.
  • The number of people you're sending the message to

Between friends, you'd probably want the cost of messaging to be extremely low, and in a situation where the correspondence factor is high, the cost would be expected to average out. In this situation, it would be important to have extremely low transaction fees, or to have a way to batch transactions (ie level 2). For ETH, this would probably be a level 2 protocol, though if a fast and cheap network like Avalanche is used, it could be done on-chain.

Note that this would be real wealth being transferred between people, alongside information. You'd literally be paying people to read your messages. The pricing mechanism would have to be designed to make it totally cost-prohibitive to spam people.

It's like the traditional post system, except that we pay each other instead of a centralised third party. We pay each other to contribute and act as valuable citizens.

Also, there could be many networks, and each network could have different rules. So, for example, you could have one b-mail for personal correspondence, and another for newsletter-style correspondence, and the pricing mechanisms could be quite different. Each network has a different set of smart contracts, built on the same fundamental mechanism.

Reputation

Why is reputation needed? Why can't we just use money? Because most games humans play are about winning either wealth or status, and we tend to trade one for the other. Blockchain communities are necessarily collaborative, and reputation is extremely valuable in a collaborative society.

How does one earn reputation? By sending emails which are not marked as spam.

In a system where we want to incorporate some kind of reputation system, after reading the message, the recipient could publish an acknowledgement on the chain with an indication of how valuable the information was. This could be as simple as yes/no/abstain. There would probably need to be rules here to prevent gaming the system. For example, the weight of the acknlowedgement could decrease over time. The pricing mechanism could then be based on your reputation.

The reputation would be stored in a token, and a set of smart contracts would goven their behaviour.

Reputation could be fungible (ie if I have 10 reputation, I can "endorse" someone else up to 10-N reputation, N tbd, for example - there could be other endorsement rules). This could permit anyone to create multiple pseudonyms which, with some ZKP magic, all "share" their reputation, but cannot be linked. This could even work across networks.

Issues/Questions

Reputation systems are hard, and this is vulnerable to the classic average rating problem, where one extremely negative review ("you killed my friend") is lost in a sea of mediocre reviews ("I use this sometimes, works ok")? (XKCD ~ TornadoGuard)

What does having more reputation "buy" you? Other than cheaper messages? Maybe there could be rules that allow people to block messages from senders with reputation below a given value.

Why does this work?

As a recipient, if you value the information, you're incentivised to not mark it as spam (because if you do, you're less likely to receive more of it).

As a sender, you're incentivised to think carefully about the information you send, because it costs you real money to send it. The meaning of "think carefully" would depend on the network rules as described above, and it could, for example, involve some kind of reputation consideration.

If (and only if) the rules are chosen carefully, and assuming there is an equilibrium condition (that is, the cost of messaging is stable in some sense), the net effect is that the information transferred is mutually valuable to both the sender and recipient, and everyone is happy.

Of course, this system doesn't feel at all necessary for communicating between friends, and it seems likely that email could continue to be the protocol of choice there. However, as a mechanism of communication between organisations, or as a mechanism of news-distribution, this could be nice.

Appendix

A deep problem with blockchain: how to change applications after they've been deployed.

  • One general solution might be a kind of contract which automatically transfers wealth (eg COIN1 -> COIN2) when certain conditions are satisfied. But is there a Gödel loophole here where the master contract itself can be changed (maliciously) to extract wealth?

What does decentralised mean? Ethereum is a platform for hosted protocols (smart contracts). The current internet is a home for applications. Protocols are valuable because anyone can participate without needing to obtain consent from a central party.

2020: A Year Of Serverless

February 2021 Draft - still thinking about this.

I spent most of 2020 working with Serverless systems, and this is a summary of my thoughts and findings.

Here's a controversial statement: cloud engineers are not programmers. The industry (including me) is building cloud applications like hardware, not software. Most cloud teams have a heavily waterfall (!) style process, though they won't admit to it.

There's a requriements gathering phase, a design phase, an implementation phase, and a test phase (all of this usually spread over several weeks). Care is taken at every step to plan things because infrastructure, like hardware, is hard.

Shouldn't we aspire to treat cloud applications just like real software?

Serverless technology (cloud functions, not platforms) is moving us in the right direction, but isn't there yet.

Microservices are an anti-pattern?

Why do we use microservices?

  • Simple replacement of service components (deployment)
  • Separation of concerns
  • Multiple programming languages in one system
  • Other reasons?

Problems they cause:

  • Rigidity and brittleness, leading to up-front design work.
  • Testing complexity
  • Waterfall-style development (services, agree on apis, implement, integrate, test).

Microservices done badly are just distributed monoliths, which are 10x worse than normal monoliths, because network unreliability.

Are they an anti-pattern which only exist because current deployment methods aren't good enough, and the application abstractions are leaky? Maybe we formally optimise structure because there isn't a compiler which can optimise it for us).

Could we apply traditional software engineering approaches to cloud systems?

For example, if we had a C/C++ program with multiple components, we would compile each component individually, and then link them together to form the complete application. We could let multiple teams work on the complete application - each team builds libraries (microservices).

If our cloud deployment system was smart enough, it could iteratively deploy the application, so we eliminate all the reasons to use microservices (except the multiple languages reason).

Serverless - Soft Infrastructure

There aren't that many ways to build serverless projects, Serverless Framework is great for small projects, but if you're doing something bigger, you'll need to understand AWS.

Cloud functions are probably commodities -- most cloud providers have them and prices are being driven downwards and performance is shooting upwards. This is great for us, so I spent a big chunk of 2020 (during the Covid-19 lockdowns) working on a different approach to serverlesss computation.

The key goal was to take the orchestration logic out of the programmers hands, so that a "compiler" can determine the optimal infrastructure and glue logic. Just like a C/C++ compiler determines the optimal sequence of CPU instructions. The novel contribution was to allow the programmer to use cloud functions as threads in a concurrent programming language. This is then general and poweful enough to implement any kind of workflow.

The implementation is a runtime (virtual machine) and a compiler. The runtime defines semantics for concurrent computation distributed across multiple threads. The compiler maps programs into infrastructure for specific cloud providers.

Secondary goals:

  • Development speed
  • Portability - as the programmer does not directly specify the infrastructure, one could define compilers and runtime environments for any cloud.
  • Zero infrastructure management - programmers shouldn't be doing infrastructure, because infrastructure is hardware.

Result: Hark.

Some questions

Is it better to build a new language and VM, or implement a cloud-native runtime for an existing language/VM (eg Python or Go)

C is heavily designed for a specific CPU architecture, which is precisely what allows you to build performant programs in it. Why should cloud be different?

Go might be "trivially" portable.

Is a turing complete language really necessary to describe the kind of

systems we implement in the cloud? Or is a workflow language sufficient (ie, Airflow, Dagster et al)?

My gut is that programmers using workflow-style systems eventually try to write "proper" programs in the workflow style, because that's what programmers do.

Could we build something like Hark on top of Firecracker to orchestrate functions?

This could reduce the startup overhead...

But it'd be a new platform, and would compete with AWS Lambda. Probably not a great idea.

How do we make data sharing performant?

Storage and sharing data across threads is slow, because there's network traffic and we can't control the placement of our containers in AWS's infrastructure. Are we just waiting for someone to build a better database? (distributed database optimised for specific workloads).

Reading Material

Quite a few institutions and clever people are thinking about ways to formalise serverless computation. This is great!

Formal foundations of serverless computing. 
Jangda, A., Pinckney, D., Brun, Y., & Guha, A. (2019). Proceedings of the ACM on Programming Languages, 3(OOPSLA), 1–26. https://doi.org/10.1145/3360575

  • Great academic paper which lays out a formal lamdba calculus for distributed computation across cloud functions, encapsulating the execution semantics (retries, etc).

Formalizing Event-Driven Behavior of Serverless Applications. 
Obetz, M., Patterson, S., & Milanova, A. (2019). http://arxiv.org/abs/1912.03584

  • Another nice formal definition of operation semantics for serverless systems.

The Stanford Builder (GG). https://www.usenix.org/system/files/atc19-fouladi.pdf

  • Some great work at Stanford. The ffmpeg demo is incredible.

The Design of Stateful Serverless Infrastructure. https://www.vikrams.io/papers/thesis.pdf

  • A completely new platform for stateful "serverless" computation. Very nice work; bold move to entirely replace AWS Lambda / GCF.

Serverless: I’m a big kid now. https://www.simplethread.com/serverless-im-a-big-kid-now/

  • A great summary article of serverlesss in 2020.

Build What They Want

January 2021

TL;DR: I spent years mistakenly thinking I understood what Paul Graham meant. I'm trying something slightly different now.

I prefer a different wording of Paul Graham's advice for startups.

Build What They Want makes the precedence obvious: before you start, you have to find out who they are, and what they want.

It’s not about building what you want.

Paul Graham starts with “build something”, and that’s what hits your subconscious first, allowing the “something” to remain ambiguous. I'd prefer to start with “build what”, so you’re forced to evaluate the “what”. The original wording lets you think that you can build something and convince people to want it later (once you show them and tell them how great it is, and all the important features it has...).

This changes everything.

My problem is that I find it too easy to imagine things that I could build, and once I’ve imagined something, I struggle to not want to build it exactly that like. Because I enjoy the process of building.

Building is a wonderful thing to enjoy. If you want to start a businesss, just do it differently.

Instead of applying imagination to products, apply imagination to companies. What company could exist if someone (you) went and built it? What excites you? A company is built on something that people want.

How do you know what people want?

Talk to people. Understand them. Ask questions.

Choose who you talk to. Talk to people who you think might want something that you would be excited to build. Don’t imagine the product though. Imagine the kind of change you could make in their lives.

They probably won’t be able to describe exactly what they want; the skill you must develop is in spotting the unexpressed (or inexpressible) want.

But whatever you do, your goal must be to build something they want.

This is strongly related to jobs to be done. The things they want will always be related to the things they already do. Find that link.

Last year I built something that I wanted to build. I only partly wanted the final product. Mostly, I wanted the experience of building it. And actually that turned out to be extremely valuable, though it wasn’t the right way to build a business.

Then again, it’s important to geek out and build interesting things without immediate value. Contribute something novel, and maybe one day it will be useful to someone else (even just as prior art).

And if you’re going to build something for the love of building, you should definitely do so in public. You’ll make new friends and learn much more. So publish your code, get a website up, and tell people what you're doing.

You can start by telling me.

Send feedback and thoughts: Twitter // ric@rmhsilva.com

Read more: Blog archive // Book notes // RSS