Machine Learning on a breadboard

[jcpst]

I thought this was pretty cool. Analog AI algorithms!

An Analog Network of Resistors Promises "Machine Learning Without a Processor," Researchers Say

Prototyped on a series of breadboards, this analog machine learning network could one day deliver more energy-efficient AI.

www.hackster.io

 

[RetiredUnit1]

I thought this was pretty cool. Analog AI algorithms!

An Analog Network of Resistors Promises "Machine Learning Without a Processor," Researchers Say

Prototyped on a series of breadboards, this analog machine learning network could one day deliver more energy-efficient AI.

www.hackster.io

A little light on the details, as someone who spent a career programming there was really no mention of how it learns, how to teach it, how it remembers, how it is implemented.

Skeptic? Of course I am, I'm a programmer. You know how many people I've had lie to my face while I held a piece of paper in my hands with their login ID on it? Let me just count, oh yeah, it was *everyone*

 

[jcpst]

Fair point. The article doesn’t get deep. I think skepticism is a healthy perspective with any of this stuff. I’m a programmer as well, and I work with integrating custom ML models into business applications.

Here’s a keynote by the project lead that gets into more details about the work:

I was just thinking about if there’s new applications that use analog circuits, it could provide some sort of benefit to analog tinkering elsewhere.

 

[RetiredUnit1]

Fair point. The article doesn’t get deep. I think skepticism is a healthy perspective with any of this stuff. I’m a programmer as well, and I work with integrating custom ML models into business applications.

Here’s a keynote by the project lead that gets into more details about the work:

I was just thinking about if there’s new applications that use analog circuits, it could provide some sort of benefit to analog tinkering elsewhere.

There's kind of a blurry line here. There's really no such thing as a "digital" component. Computers are made from resistors, capacitors, transistors. The "digital" part is the emulation. The components are analog tho.

I took programming classes in the 70's and have built rudimentary "digital" circuits using a breadboard made up of a clock, and flip-flops and light bulbs. Got extra credit for building a 'square rooter'.

The emulation is that a flip flop in one current state is a zero and a the other current state is a 1. Off and On as it were. From this massive amounts of encoding follow. In other words how many flip flops constitute a chunk of data. My PC is 64, at the time I left college most computers were 8, lol.

So digital is an emulation of an encoding scheme implemented on analog circuits.

 

[jcpst]

Also a fair point- the line is blurry at that level.

But the line is distinct at the interface level. The way you interface with a digital device is with discrete values, while an analog device is working with continuous values.

But I think what they are saying is plausible. I could imagine how it might beneficial to have transformer layers with an analog interface.

Before we had fast digital processors, the military used analog computers to calculate a missle’s trajectory from a moving ship. Analog computers did this efficiently, in real time.

It’s a similar problem space mathematically- all those vector calculations in LLM transform layers.

 

[giovanni]

I didn’t read the whole article but my understanding was that they would take advantage of the non linearities of analog circuits to implement neural networks which would save a lot of computational power spent with the encoding as @RetiredUnit1 mentions. That seems like a very neat idea. I’m curious to see if it actually works.

 

[jcpst]

thank you giovanni- that was going right over my head!

 

[RetiredUnit1]

I didn’t read the whole article but my understanding was that they would take advantage of the non linearities of analog circuits to implement neural networks which would save a lot of computational power spent with the encoding as @RetiredUnit1 mentions. That seems like a very neat idea. I’m curious to see if it actually works.

Still a lot of questions. What is the interface? How to program/train? Why try to do this with physical devices when you can just implement a software model of the same interactions?

As I mentioned, digital is a virtual construct. It doesn't actually physically exist, it is a series of commands. Flip-flops can do two things, add and subtract. That's IT. Everything else a computer does is a virtual abstraction on top of that.

One of the things I found odd about spending 45 years programming is that I had no raw materials, everything I created, over 10 million lines of code, nothing physical.

Originally computer developers attempted to use 10 states, not just ones and zeros, but zero through 9 states of voltage present. It was overwhelming and failed. This would have been more analog-ish, but was impossible with tube technology.

It may have been easier to develop after the creation of the transistor, but by then working models were just transferred to the two new switching transistors that go into the creation of a flip flop, and of course 'and gates', 'nand gates' and a whole host of other boolean operands that make processing possible.

I have to admit I don't have the time to watch an hour and 21 minute video. But I was considered one of the best programmers in the unidata community. I fixed problems in 15 minutes that 12 programmers before me had spent 100 hours on and failed!

 

[giovanni]

Still a lot of questions. What is the interface? How to program/train? Why try to do this with physical devices when you can just implement a software model of the same interactions?

As I mentioned, digital is a virtual construct. It doesn't actually physically exist, it is a series of commands. Flip-flops can do two things, add and subtract. That's IT. Everything else a computer does is a virtual abstraction on top of that.

One of the things I found odd about spending 45 years programming is that I had no raw materials, everything I created, over 10 million lines of code, nothing physical.

Originally computer developers attempted to use 10 states, not just ones and zeros, but zero through 9 states of voltage present. It was overwhelming and failed. This would have been more analog-ish, but was impossible with tube technology.

It may have been easier to develop after the creation of the transistor, but by then working models were just transferred to the two new switching transistors that go into the creation of a flip flop, and of course 'and gates', 'nand gates' and a whole host of other boolean operands that make processing possible.

I have to admit I don't have the time to watch an hour and 21 minute video. But I was considered one of the best programmers in the unidata community. I fixed problems in 15 minutes that 12 programmers before me had spent 100 hours on and failed!

As I said, the reason to use a physical device is to implement the same math but at a fraction of the power consumption. I think this is at least a very interesting idea and at best it could revolutionize ML inference, especially on embedded devices but not only. And it also may end up being closer to the way an organic brain works so it’s kind of exciting. As far as the interface and training goes, this is just a prototype so I don’t think that’s the most relevant part. But, as far as training goes, I can imagine just training on regular old digital machines. The cost of training a model is dwarfed by the cost of running inferences for billions of users. Besides, if you can do a forward pass (inference), you can easily do a backward one (gradient descent). About the interface, I can imagine a design that allows you to set voltages or resistances to specify the weights of the network (that’s all you need for the forward pass). This is usually a set and forget process (until the next retrain which most large companies do every few weeks or even months for production models).

Note that I don’t have details - I also haven’t watched the long video although I’m looking forward to - but I have enough ML experience at this point that this article spoke my language enough to infer where they are going with it. And I really do think it’s a cool idea! Also note that the prototype currently is not actually low power (being made of discrete components and, well, just a prototype) but we know very well how to squeeze computational power out of circuits while minimizing power consumption. This is all exciting stuff!

 

[RetiredUnit1]

As I said, the reason to use a physical device is to implement the same math but at a fraction of the power consumption. I think this is at least a very interesting idea and at best it could revolutionize ML inference, especially on embedded devices but not only. And it also may end up being closer to the way an organic brain works so it’s kind of exciting. As far as the interface and training goes, this is just a prototype so I don’t think that’s the most relevant part. But, as far as training goes, I can imagine just training on regular old digital machines. The cost of training a model is dwarfed by the cost of running inferences for billions of users. Besides, if you can do a forward pass (inference), you can easily do a backward one (gradient descent). About the interface, I can imagine a design that allows you to set voltages or resistances to specify the weights of the network (that’s all you need for the forward pass). This is usually a set and forget process (until the next retrain which most large companies do every few weeks or even months for production models).

Note that I don’t have details - I also haven’t watched the long video although I’m looking forward to - but I have enough ML experience at this point that this article spoke my language enough to infer where they are going with it. And I really do think it’s a cool idea! Also note that the prototype currently is not actually low power (being made of discrete components and, well, just a prototype) but we know very well how to squeeze computational power out of circuits while minimizing power consumption. This is all exciting stuff!

I do agree it sounds like fun. But more an exercise in futility I'm afraid.

imnsho, fusion power will be online before this is even close to a working model. And quantum computers will be done before then as well, making current processors look like donkeys attached to a plow compared to a warp drive. We're talking the entire processing power of every cpu currently on the planet in one cpu.

Quantum computers will work by teleporting photons into registers with ZERO travel time instead of nanoseconds. Photon teleportation happened over a decade ago.

Overunity fusion became a reality in Dec 2022. It's only a matter of time now.

We'll still need oil for plastic and lubrication. But oil is not from dinosaurs. It is the same reaction that creates diamonds. The energy from the NUCLEAR CORE of our planet along with almost inconceivable pressure transforms carbon, the most common atom in the multiverse, into oil and diamonds.

The current supply without that which is still being created is good for the next 250 - 500 years by 'real' scientists (not on the payroll of mega-corporations.)