You are reading content from Scuttlebutt
@Anders

SSB meta feeds

As part of our work on the #ngipointer grant, over the coming months we are going to explore a new concept in SSB called meta feeds. The main idea is to provide some more structure to the messages of a feed. We do this by creating multiple feeds with different purposes belonging to the same identity and linking these feeds in such a way that other peers can find the data they need. It is important to note that all of these feeds will be tied to the same device in the same way that feeds are today.

I highly recommend watching @andrestaltz talk on this topic from our advisor meetings. Also the latest version of the design document for meta feeds is located here.

Concepts

Meta feed

A meta feed is feed where the messages only describe other feeds. These messages can then also contain meta data about the feeds:

{ type: 'metafeed/operation', operation: 'add', feedtype: 'classic', purpose: 'main', id: '@main' }
{ type: 'metafeed/operation', operation: 'add', feedtype: 'bamboo', purpose: 'applications', id: '@applications' }

In the example above we have two messages, one linking to our existing main feed and one linking to another meta feed for application specific data. In this way we can build up a tree structure of feeds. It is important to note that because these meta feeds are just regular feeds, the messages can be encrypted and thus a whole subtree be hidden from outsiders.

These new feeds will be generated from a seed value, using something like this:

const seed = crypto.randomBytes(32)

const prk = hkdf.extract(lhash, hash_len, seed, salt)
const mf_info = "ssb-meta-feed-seed-v1:metafeed"
const mf_seed = hkdf.expand(hash, hash_len, prk, length, mf_info)
const mf_key = ssbKeys.generate("ed25519", mf_seed)

By saving this seed it is possible to recreate all the keys as the process deterministic. Not all feeds in a meta feed needs to be generated in this way though.

Content feed

A content feed is similar to existing feeds in that they just contain messages like posts, likes etc.

Index feed

An index feed is a special feed that is used to index other feeds just like in a database. One example of this would be the contact messages of your main feed. You can create a feed that only contains hash links to the original messages on another feed. We are going to introduce a new RPC replication mechanism that will efficiently be able to replicate these feeds. The purpose of these feeds is to enable partial replication by only that specific part of a larger feed.

Identity

Currently in SSB your identity is the same the public key for your feed. This key is also used for secret handshake to establish an authenticated and encrypted connection between two peers. Furthermore this identity is also encoded in messages such as contact messages.

We aim to use the id of the top meta feed as the identity of a peer going forward. For backwards compatibility secret handshake is probably still going to use the main feed identity for a foreseeable future, but new concepts such as trustnet that will be using specific feeds for the messages, will refer to this new identity.

Potential applications

A list of some concepts that could be implemented using the meta feeds:

  • New feed formats
  • Sub feeds for specific application, like git-ssb, so only clients
    that actually want these messages have to store them.
  • Ephemeral messages by adding a lifetime to the metadata of a feed
    and having clients respect that.
  • Allow lists, meaning you can specify what other feeds are allowed
    to get a certain feed
  • Same-as using feedless identities. I'll be dedicating a separate
    post for this topic.

What we aim to build

This document describes the ideas in more detail, but I'll briefly highlight some ideas:

  • Indexes
  • Claims (indexes for other feeds) and audits using trustnet
  • Feedless identities for public small groups

Our aim with this is to get the number of messages a client needs to download down to about 1/10 of the size, which should decrease the time a new user needs to wait substantially.

Backwards compatibility

We aim to keep as much backwards compatibility as possible with existing feeds so that existing client applications will continue to work. As meta feeds is a layer on top of existing feeds in a way it is natural to implement only new concepts using meta feeds. If it turns out to be a good idea and we have a better feed format then I think that would be the point where existing clients would need to be adapted.

@sir Moid

@arj the first example references the main feed and also an applications feed of type Bamboo. Then the claim is made that one can build a tree of metafeeds, as a metafeed can contain other metafeeds.

Does that imply the applications feed, of type bamboo, is a metafeed, or is the example just not the best? Shouldn't it be something like "feedtype: metafeed"?

@Anders

@moid you are correct that the applications feed is a meta feed with another feed format (bamboo). I'll change the name to feed format. Thanks for that feedback.

The example could have been clearer if a feedtype was added I agree, yes. I'm just wondering if we need to specify that up front or if it can be deduced from the content.

@sir Moid

@arj

I guess it can always be deduced from the content, as all messages in a metafeed have a "type: metafeed/operation" or "type: metafeed/..." ? Is that the convention?

@Anders

@moid exactly. Where it might make sense to know the type up front is for index feeds. So that is very good feedback, thanks. Will have to think it over.

@Rabble
Voted this
@sir Moid

@arj

one more question. Indexes are a type of feed that will support defined subsets of feeds as well as tangles, thereby providing partial replication. Very cool.

With respect to tangles will the messages in the index only contain cypherlinks, or could they possibly reference the feed id as well, assuming that tangles for example would commonly involve several feeds? If only the cypherlinks, then the assumption is that there is some global database or other global state that allows one to map a message id to it's feed.

Thanks for documenting all this, and the video @andrestaltz pushed out there. I always need to hear thing multiple times for it to sink in.

@Anders

@moid right, I didn't explain that very well I see. The syntax for indexes would be like this:

{ type: 'metafeed/operation', operation: 'add', feedformat: 'classic', id: '@index1', query: 'and(type(contact),author(@main))' }

So it would be specified in the meta feed what the index covers.

Tangles are a bit of a special case. If you want to exchange that between two peers then I think a better option would be to use set replication. I did a small talk for our advisor meeting about tanglesync and also wrote down some thoughts here.

@Anders

Some updates on this. Had a call with our security audit team (@keks & justin) and we discussed using a special feed format for meta feeds. Meta feeds needs to be stable as it can outlive the main feed allowing feed rotation. By using a different feed format than ssb classic we open up the possibility of implementations with only binary feed formats. Also spec'ed a mini query language to be used for describing the contents of index feeds.

User has chosen not to be hosted publicly
@Anders

@punkmonk correct to the last point. ssb-identities don't have any relationship between feeds (on purpose). Meta feeds is meant to be on a single computer to make it easy to split data for different purposes into multiple feeds. For that purpose you then don't need to use ssb-identities.

Join Scuttlebutt now