Progress in the Analysis of Cortical Circuit Implementations

Some time ago I coined the term Discrete Cortical Circuits and described the distinguishing characteristics that many have in common. My original list of DCC characteristics included the following:

• neuron activations are binary
• synaptic weights are binary
• winner-take-all activation strategy for neurons
• input data must be encoded to binary

You can keep adding more DCC characteristics to the list:

• Hebbian learning
• high-dimensional (1000s of bits)
• sparsity (< 20% of bits active)
• structure in the binary codes (order, coordinates, topology, groups)

I’ve selected and been analyzing, what I consider, four different implementations of DCCs. None of these four satisfy all the characteristics above. However, the most interesting thing about these DCCs is in how they violate the criteria and what that changes for the performance of the algorithms. The four different DCCs I considered are: HTM, Sparsey, SPH, and BrainBlocks.

I had vague ideas about what i wanted to accomplish analyzing and comparing these architectures. I was hoping that I would somehow find the core commonalities or essential features that would enable a new fundamental understanding of intelligence. I’m not sure I’m there yet, but I certainly have discovered some knowledge that I hope to share with you.

One of the things I’ve been building is what I call the DCC Reference Model which is accompanied by a taxonomy of architectural decisions you can make. An example of what I mean by this would be, deciding how you are going to select which cells to activate and in what structure they can be broken down by:

• grouping: partitioning your cells into groups or keeping them unpartitioned (one group)
• winner-selection: a single winner per group, k winners per group, or at-most-k winners per group.

There are many other architectural facets, but I will come back to them later.

Another thing I wanted to do was build a common interactive sandbox so that people could play with these algorithms, easily assemble them, scale to extremely large networks, and visualize their operation. For that purpose, I’ve been working on my DCCcore framework which is something hasn’t been released yet. Not only does it have a high-performance Rust backend, but I have created a browser-based frontend. You can now run DCCs in your own browser with amazing speed and efficiency. They also provide real-time graphics and interactivity.

I initially announced my web-based dashboard project on my blog. Since then, I have been posting new iterations of my visualization and interactive applications on my DCC public repo. No source code for these yet, but the WebAssembly apps can be embedded in any web page or viewed directly. Feel free to check out some of my older web demos.

Anyway, as part of the effort for the comparison and analysis of other DCC architectures, I ended up porting the existing implementations from their C++ or Java languages to the Rust language. This not only lets me embed them into the browser with WebAssembly, but it also helped me understand the different DCC architectures better. I definitely had some mechanical help doing this with the aid of your friendly neighborhood LLM agent. One of the consequences of this is that the resultant product is heavily annotated and documented to ensure clear human and machine understanding of the code, with a set of standards and tests to ensure fidelity to the original project. You can dive into the doc subdirectories to look at the reframing of these projects in words other than the developers. I think even this latter part is very useful since developers often have a hard time explaining their own work since they’re so deep in it. I know I have this problem too!

I have released the source code and repos of the compared DCCs. Though, I was the co-developer of BrainBlocks, I haven’t released the ported Rust code since there are some IP issues I need to clear with the company I worked at when writing this. Other than this, I have used the licenses provided by the project or got written permission where the reference code was not publicly available.

Here are the three repos I have released:

• dcc-htm — Hierarchical Temporal Memory (HTM) from Numenta’s htm.core (bef3088)
• dcc-sparsey — Sparsey, from Gerard Rinkus’s SparseyCore (a0d4d34)
• dcc-sph — Sparse Predictive Hierarchies (SPH) from Ogma’s AOgmaNeo (645a54a)

The original reference implementation for HTM is C++ with Python bindings, for Sparsey is Java, and for SPH is C++. The Rust repos are fully stand-alone implementations with example code, documentation, and tests provided. You may encounter the occasional error or AI hallucination, but I’ve inspected the work myself and enforced a lot of quality on the end product.

I would welcome anyone who dives into these projects to offer any corrections or improvements you might have. I’d also welcome any further example code or implementation of features that I didn’t provide. The repos have to conform to a few minimal structural standards so that they can be fetched and built with my framework and web dashboard. Other than that, these projects stand on their own.