nyx2d
software engineer with an interest in decentralization
software engineer with an interest in decentralization
this post is a work in progress. please note that in code snippets I have removed error handling for the sake of brevity.
I have long known that I'm a tactile learner when it comes to software. I never feel like I truly understand something until I've built (or attempted to build) a prototype or proof of concept. at this point in my programming career, I probably have hundreds of unfinished projects on my drive. completion is never the point - I do enough to get some critical concepts to stick and then move on.
I've been fascinated by p2p technology for a long time. within the last few years I've been keeping my eye on several cool projects, especially things like ssb, earthstar, hypercore, and p2panda.
I've hacked on a number of p2p adjacent things over the years, including a (very) small social network that works over WebRTC and uses public/private key pairs for identity. still, I had never really gotten my hands dirty with lower level implementations. this is a quick and dirty log of my process doing that.
for legible IDs I decided on a simple base64 encoding.
pubKey, privateKey, _ := ed25519.GenerateKey(rand.Reader)
peerID := base64.StdEncoding.EncodeToString(pubKey)
next up was local peer discovery. I considered a few options here, including rolling my own UDP broadcast based system, but ultimately decided to use mDNS alongside DNS-SD. as it turns out, this is also the approach earthstar is working on.
there's already a great go library by hashicorp focused on this exact use-case (mDNS+DNS-SD). it was very easy to setup service advertising.
func Broadcast(peerID string, servicePort int) {
service, _ := mdns.NewMDNSService(peerID, "my_network", "", "", servicePort, nil, []string{peerID})
server, _ := mdns.NewServer(&mdns.Config{Zone: service}) // kicks off goroutine in constructor
log.Printf("broadcasting as peer %s\n", peerID)
}
to enable discovery on the other end, I wrote a simple function that preforms an mDNS query every few seconds. I spawn a goroutine to run this indefinitely.
const period = 10 * time.Second
const timeout = 2 * time.Second
const bufferSize = 256
func FindPeers(peerID string) {
ticker := time.NewTicker(period)
for ; true; <-ticker.C {
entriesChan := make(chan *mdns.ServiceEntry, bufferSize)
mdns.Query(&mdns.QueryParam{Service: serviceName, Domain: "local", Timeout: timeout, Entries: entriesChan, DisableIPv6: true})
entries, _, _, _ := lo.BufferWithTimeout(entriesChan, bufferSize, timeout)
for _, entry := range entries {
addr := &net.TCPAddr{IP: entry.AddrV4, Port: entry.Port}
if entry.Info != peerID {
// handle newly discovered peer
}
}
}
}
I know there's probably a cleaner way to write this logic but hey, it works
after discovery, my next step was deciding on the actual communication protocol. I considered a few options, including rolling something from scratch using UDP (to more easily support NAT traversal). ultimately I decided to just use gRPC, as it's very well established and supports protocol buffers.
setting up a grpc server is very straightforward. you first need to define a protobuf file that defines the RPC interface. you then run the protobuf code generator, which outputs Go.
message Message {
oneof content {
SignedMessage signed_message = 1;
}
}
service Network {
rpc Gossip(stream Message) returns (stream Message);
}
you then need to implement the server interface. I decided to use a single rpc call with bidirectional streaming in an attempt to dissolve the distinction between client and server gRPC implicitly has.
type server struct {
pb.UnimplementedNetworkServer
}
func (s *server) Gossip(stream pb.Network_GossipServer) error {
for {
val, err := stream.Recv()
if err == io.EOF {
break
}
// handle message
}
return nil
}
to actually register your server instance, you need to bind it to a TCP socket. this is also where we get auto assigned port info to pass to our earlier broadcast function.
func Serve(peerID string) int {
lis, _ := net.Listen("tcp", ":0")
pb.RegisterNetworkServer(grpc.NewServer(), &server{})
go func() {
log.Printf("gRPC server listening at %v", lis.Addr())
s.Serve(lis)
}()
return lis.Addr().(*net.TCPAddr).Port, nil
}
while we know what peerID a node is advertising as from their DNS-SD Info data, we don't currently have any way to validate that, nor ensure that the traffic we're receiving hasn't been tampered with by a man in the middle attack.
so to address this I decided to tackle authentication. gRPC supports a few authentication schemes, notably TLS. unfortunately, gRPC's implementation only supports the standard x.509 certificate scheme, which usually comes with the implication of a centralized certificate authority.
at first I decided to leverage self-signed certificates using the peer's ed25519 keys instead, but this proved more complex and less interesting than I would have liked.
another solution is to convert the peer's ed25519 key into x25519 key (following something like this blog post and this library). the x25519 key can then be used with the crypto/ecdh package to generate a shared secret that can be used to encrypt traffic.
a third, and the ultimate solution that I settled on, is to generate generate temporary x25519 keys for each peer connection, and to preform a elliptic-curve diffie-hellman key exchange to generate a shared secret. the public keys are each signed with the peer's long term identity key to prove ownership.
here's the simple exchange I decided on:
now that each client has a shared secret, they can use this for encryption between them.
it was at this point I grew frustrated with gRPC's client/server distinction, as I realized I would basically need to roll my own RPC protocol inside of the proto scheme. I decided it would be more fun to roll my own from scratch without gRPC's baggage. thus, I decided to pivot to using a simple CBOR based protocol directly over TCP instead.
func (c *Conn) SendMessage(m *Message) {
data := cbor.Marshal(m)
binary.Write(c.conn, binary.LittleEndian, uint64(len(data)))
c.conn.Write(data)
}
func (c *Conn) ReadMessage() *Message {
sizeBuf := make([]byte, 8)
io.ReadFull(c.conn, sizeBuf)
size := binary.LittleEndian.Uint64(sizeBuf)
readBuf := make([]byte, size)
io.ReadFull(c.conn, readBuf)
var m Message
cbor.Unmarshal(raw, &m)
return &msg
}
the protocol is exceptionally simple, just a little endian uint64 length prefix to a CBOR encoded message struct. encrypted messages use the same scheme, and just dump the CBOR binary blobs into a wrapping object. the key exchange works through the same protocol.
type Message struct {
*Handshake
*Encrypted
}
type Encrypted struct {
Payload []byte // encrypted CBOR encoded message
}
type Handshake struct {
SigningPublicKey []byte
TransportPublicKey []byte
Signature []byte // sig of transport_pubkey by signing_pubkey
}
under the CBOR-based protocol, the key exchange is implemented as follows.
we track peers in this EncryptedConn struct:
type EncryptedConn struct {
*Conn
// our temp keys for transport encryption
transportPublicKey *ecdh.PublicKey
transportPrivateKey *ecdh.PrivateKey
peerTransportPublicKey *ecdh.PublicKey // their temp public key for transport encryption
peerPublicSigningKey *ed25519.PublicKey // their long term public signing key from the handshake
// our shared secret
transportSharedKey []byte
transportCipher cipher.AEAD
}
for each new peer, we generate a temporary pair of "transport keys" that will be used for wire encryption.
func NewEncryptedConn(conn net.Conn) *EncryptedConn {
transportPrivateKey, _ := ecdh.X25519().GenerateKey(rand.Reader)
return &EncryptedConn{
Conn: NewConn(conn),
transportPrivateKey: transportPrivateKey,
transportPublicKey: transportPrivateKey.PublicKey(),
}
}
handshakes are carried out as follows, with "client" or "server" only meaning who established the connection first.
func (c *EncryptedConn) HandshakeAsClient(signingKey ed25519.PrivateKey) error {
c.Conn.SendMessage(c.buildHandshakeMessage(signingKey)); err != nil {
serverHandshake := c.Conn.ReadMessage()
if !c.verifyHandshake(serverHandshake) {
return errors.New("handshake verification failed")
}
c.deriveSharedKey()
return nil
}
func (c *EncryptedConn) HandshakeAsServer(signingKey ed25519.PrivateKey) error {
clientHandshake := c.Conn.ReadMessage()
if !c.verifyHandshake(clientHandshake) {
return errors.New("handshake verification failed")
}
c.Conn.SendMessage(c.buildHandshakeMessage(signingKey))
c.deriveSharedKey()
return nil
}
building the handshake message is straightforward, as is verification
func (c *EncryptedConn) buildHandshakeMessage(signingKey ed25519.PrivateKey) *Message {
sig := ed25519.Sign(signingKey, c.transportPublicKey.Bytes())
return &Message{Handshake: &Handshake{
SigningPublicKey: signingKey.Public().(ed25519.PublicKey),
TransportPublicKey: c.transportPublicKey.Bytes(),
Signature: sig,
}}
}
// verifyHandshake mutates the conn to include the peer's public keys
func (c *EncryptedConn) verifyHandshake(m *Message) bool {
peerTransportPublicKey, _ := ecdh.X25519().NewPublicKey(m.Handshake.TransportPublicKey)
ok := ed25519.Verify(m.Handshake.SigningPublicKey, m.Handshake.TransportPublicKey, m.Handshake.Signature)
if ok {
c.peerTransportPublicKey = peerTransportPublicKey
c.peerPublicSigningKey = (*ed25519.PublicKey)(&m.Handshake.SigningPublicKey)
}
return ok
}
deriving the shared secret post handshake is similarly straightforward
func (c *EncryptedConn) deriveSharedKey() {
shared, _ := c.transportPrivateKey.ECDH(c.peerTransportPublicKey)
cypher, _ := chacha20poly1305.NewX(shared)
c.transportSharedKey = shared
c.transportCipher = cypher
}