Artificial Intelligence

Learning occurs by adjusting the weight matrix in a neuron

Care to explain how? The logic required to update the weights is generally-speaking exponentially higher in computational cost than the actual neuron firing logic / dynamics, which is quite trivial to compute, even on an ordinary PC. One of the big problems with ANNs (Artificial Neural Networks, which is what you are describing, and btw, you should not use the general term AI as being synonymous with ANNs, because ANNs are a very tiny subset of AI) is the fact that algorithms to learn the weights of an ANNs are either (1) computationally intractable, (2) highly undercontrained, or (3) very crude and thus requiring a life-time of learning (which is the way animal brains learn).

The forward-eular trick can be used to approxiate exponential decay thereby reducing the calculation computations.

It seems to me like the whole thing could be done with an analog computer. This whole thing, in fact, looks like the dynamics of a fairly ordinary switched-capacitor circuit. I would imagine that with a few op-amps and transistors, this could be realized quite easily. If you need fast and instantaneous simulation of some dynamical system, analog computers are your friend, they've been used quite a bit in robotics to do some of the heavy calculations instantaneously. In fact, why not try this with a FPAA?

Also, if you are going to using forward-Euler, you might as well just stop thinking in terms of a continuous-time system, because it's not anymore. Discrete-time systems and discrete-time dynamics is, in my opinion, a much better way to analyse systems anyways, so you might as well go for that. I have found that systems can be very greatly simplified if you do the analysis entirely in discrete-time, instead of using something like Euler method to discretize it only at the end (which can also lead to some bad surprises).

In any case, from what I know, there is a pretty good concensus in the AI field that the actual mechanics of the neuron firing is not really relevant to the overall schemes (learning, performance, "intelligence", etc.). In fact, modern research in ANNs has departed quite a lot from anything that actually reproduces the exact dynamics of neurons, mostly because it's too complicated and it has been shown to provide no benefit at all, it's just one of these things where "nature" did it that way because that's just how it happened, but there are better and simpler ways.

Drawing from your analogy with Man's attempts to fly, reproducing the actual mechanics of neuron firings is the equivalent of flapping wings, i.e., it's trying to mimic nature while missing the principles that make it work. It's when people understood how lift was generated that they realized that the flapping wing designs were just overly complicated and that it did not make practical sense to reproduce that, so, instead they realized that they just needed wings with the proper shape to generate lift, and some means to propel the plane forward, and that's how the design was made.

the above equation is, in my opinion, of great importance to the field of AI.

I remain unconvinced. And, of course, if you ask what is the most important equation in the field of AI, the answer can only be Bayes Theorem, because any typical analysis of some AI method involves applying this equation about a dozen times over, it's the lens with which we see the world, and it's the single equation that defines intelligence. And the human/animal brains are just very efficient Bayesian inference approximators.

And given the vast number of basis functions (called "kernels") that are used in machine learning, whether it's for ANNs or support vector machines, you can hardly say that one is particularly more important than another. Typically, ANNs or SVMs are tested with several different kernels to find the best one to use, and the one you showed is not particularly great.

I'm curious as to why you selected a kernel that is exponentially decaying with time. Is there are reason why this time-component is important? We know that one of the limitations of neurons in an human/animal brain is the fact that they are very slow to respond (fire) due to there inherent electro-chemical dynamics, as evidenced by that graph that shows a 2.4 micro-seconds response-time, which is extremely slow from a computational point of view (which is in the order of nano-seconds). Do you think that this slow response has benefits? If so, which benefits? Why not just idealize the neurons into having an immediate response? I have some ideas why that could be important (e.g., Hebbian learning, Hopfield nets, etc.), but I'll let you develop on that.

http://www.eetimes.com/document.asp?doc_id=1320753

Why do you talk about modeling ANNs in FPGAs, and then link to an article about running some minimax routines on an FPGA? They are both methods under the very wide umbrella of "artificial intelligence", but they are very different and completely unrelated methods. They are not even in the same category of problems, as in, ANNs are traditionally for unsupervised learning / reinforcement learning, while minimax is in decision theory (finding optimal path through a decision tree).

Thanks for the reply, let me just start of by saying I'm in no way qualified to answer some of your questions. It's way over my head.

I've just put together some thoughts/ideas when I looked over at modelling neurons in the brain.

So I'd like to pose some questions of my own.

In fact, why not try this with a FPAA?

Never heard of an FPAA can you explain. I just googled an article which cites

Field Programmable Analog suffers from serious noise and performance issues because the routing fabric contains a large number of transistors.

http://electronics.stackexchange.com/questions/25525/are-there-any-analog-fpgas

as being a major disadvantage.

In any case, from what I know, there is a pretty good concensus in the AI field that the actual mechanics of the neuron firing is not really relevant to the overall schemes (learning, performance, "intelligence", etc.).

Ah interesting take on things. Can we assume that modeling the neuron more closely and more accurately will make any difference to our understanding of intelligence? Or like you said is this unimportant or are we wasting our time. Until we cross that frontier I guess we don't know for sure.

Let's ask another question, our brain is considered having the capability of consciousness. Can an electrical system man made system ever be considered conscious? After all aren't the electrical signals that fire in the brain some sort of transistor.

What is the difference between biological transistors and electrical.

what are your thoughts on this?

Drawing from your analogy with Man's attempts to fly, reproducing the actual mechanics of neuron firings is the equivalent of flapping wings, i.e., it's trying to mimic nature while missing the principles that make it work. It's when people understood how lift was generated that they realized that the flapping wing designs were just overly complicated and that it did not make practical sense to reproduce that, so, instead they realized that they just needed wings with the proper shape to generate lift, and some means to propel the plane forward, and that's how the design was made.

Ah yes that is very true. But what happened when we tried to model the complex motion of flight by 'flapping the wings?'

http://www.youtube.com/watch?v=Fg_JcKSHUtQ

I'm sure you've seen this. Flight is possible without 'flapping' as in the airplane, but by our understanding that we can mechanically model lift by flapping of the wings unlocks another door of unexplorered possibilities.

Who is to say that the importance of modelling neurons as close to our brain as possible is largely unimportant?

What is your take on this?

Why do you talk about modeling ANNs in FPGAs, and then link to an article about running some minimax routines

Good point, I guess I was just trying to highlight the performance increases as opposed to the learning method.

Slightly off topic I also wanted to link to another really interesting article which I thought was a good read.
http://www.theatlantic.com/magazine/archive/2013/11/the-man-who-would-teach-machines-to-think/309529/

Never heard of an FPAA can you explain.

I had never heard of FPAA either, until I did a little looking up of some practical ways to implement analog computations. From what I gather, they can be used like FPGAs, but instead of programming digital logic gates, you program analog circuits. It seems like a nice way to make an array of analog computers in a small and efficient package. I can certainly imagine that there would be a lot of noise generated by all those transistors being packed on a single IC. But for an application like this (modeling neurons) the noise is probably inconsequential. If you were designing analog signal processing (which is generally aimed at removing noise or demodulating signals), FPAAs would not be suitable due to the noise level. But with neuron emulation, it is a kind of pseudo-digital system (due to the thresholded firings), meaning that the noise is not going to be a factor.

The key thing to remember with "the brain" is that it is a big approximation calculator. In other words, everything approximate, signals are fuzzy, and results are probabilistic. So, a digital logic system (like FPGAs or CPUs), whose main characteristic is the ability to make exact calculations, is an odd fit in the midst of it all.

Can we assume that modeling the neuron more closely and more accurately will make any difference to our understanding of intelligence?

Does your understanding of transistors help you understand how complex software is written? (rhetorical)

That's basically my take on this debate.

Until we cross that frontier I guess we don't know for sure.

As far as I know, this frontier has been crossed a while ago. I remember writing a neural network simulator when I was still in high-school, and at that, just reproducing work from decades before. There is a huge amount of work that has been done on many aspects of this, from closer to the neuron firing mechanism (like you described) to various network topologies and learning algorithms.

But in the realm of studying actual brains and how they work, one of the promising frontiers that is starting to be tackled is the ability to map out the entire set of connections in a brain and understand how the network as a whole functions. This is starting to become technically possible (map out neural connections). That is going to be VERY interesting.

But with the years of research on ANNs, things kind of went up and down a few times. At first, there was enthusiasm for reproducing the brain mechanisms with high fidelity, but it quickly became apparent that it was at best able to perform very trivial tasks in a terribly inefficient way, so it was almost abandonned as a result. Then, people started to apply them to more appropriate problems (sort of "fuzzy logic / control" problems or optimizations) and things picked up again, especially as people took out the high-fidelity neuron-firing mechanisms and replaced them by simpler and more efficient basis functions or kernels. But still, performance could not meet its competing approaches for these problems. And finally, the work in machine learning has come to be able to understand and explain what makes neural networks so powerful (in the brain), but still, ANNs are not the best expression of these mechanisms. Nevertheless, that understanding is starting to bare enough fruits in neural networks that people are starting to make serious attempts at making an "artificial brain".

But to me, one thing is clear, the actual firing mechanism is not particularly important. It's just the mechanism that happened to occur through natural evolution.

Can an electrical system man made system ever be considered conscious?

I see no reason why they would not be able to.

After all aren't the electrical signals that fire in the brain some sort of transistor.

Well, not that simple, but in essence, it's a simple electrical device, not unlike something you could make in a semi-conductor substrate. What is most challenging is the multiple input and multicast output nature of them, which is also a hint at where the actual complexity lies (as I already explained), that is, in the connectivity and complex interactions between them, not so much within them. As the IBM researchers clearly say, for example, is that the big challenge with creating a "brain chip" is routing all those signals (synapses) not making the artificial neurons.

Another interesting thing that makes a brain different from a hypothetical electronic system is the actual brain chemistry. We know that certain chemicals have effects on brain activity and it manifests itself in interesting ways like moods and concentration. I would imagine that an electronic brain would be much more in a constant "average mood", and I'm not sure it could have emotions, at least not quite in the same way as humans do. That will be an interesting to see.

But what happened when we tried to model the complex motion of flight by 'flapping the wings?'

It's a technical feat and a very nice machine, no doubt about that. But you still have a very inefficient system and it requires cutting edge materials to even make it possible. In other words, people's initial assessment was correct. The point is that a flapping wing design requires a lot of energy and can only carry as small weight. In that talk, they quoted 20-80W and a total weight of about 400g, which those not even match the efficiency of a quadcopter (which is one of the least efficient flying devices that people use). Birds in nature are mostly able to fly efficiently because they know when to soar, when to ride hot air pockets, and when to flap, and flapping is the last resort.

The thing to understand is that nature is a mixed bag. Evolution continuously optimizes things, and billions of years can work a lot of miracles. However, evolution is also limited in how much it can "leap" (in fact, it cannot make any leaps), and it is always bound to modify whatever it started with. So, when evolution has to turn a clumsy lizard-like creature, called a dinosaur, into a flying animal, called a bird, it managed producing something that works pretty well, but not particular remarkable by engineering measures. On the other hand, with all the wealth of life's diversity in water (where it originated in), it created organisms, called fish, that can move very efficiently in water.

but by our understanding that we can mechanically model lift by flapping of the wings unlocks another door of unexplorered possibilities.

Yes, of course, I'm all for exploring those avenues and it certainly makes for interesting research and lessons to be learned. But it's also important to be cautious about holding biological or natural things up on a pedestal as if they had some sort of cosmic "perfection" to them.

Who is to say that the importance of modelling neurons as close to our brain as possible is largely unimportant?

It depends on your objectives. If the objective is fundamental science, then pretty much anything is important as long as it's currently unknown or poorly understood (which, besides neuro-chemistry, I don't think is the case for neurons). If the objective is applied science or engineering, such as creating some artificial intelligence or brain, then what is important is what allows you to progress towards that goal, and in that vein, I don't think that further investigations of the details of neuron firing mechanisms are a particularly practical line of investigation, because we know that the real complexity lies elsewhere, in particular the network topology and learning mechanisms.

Slightly off topic I also wanted to link to another really interesting article which I thought was a good read.

I agree, it was an interesting read (albeit long and poorly written..). Cognitive science is definitely an important field that too often gets snobbed by the machine learning people. This is also another huge polarizing issue.

The machine learning folks would generally argue that once you get the learning mechanisms working, which are at the root of intelligence, the complex mechanisms of intelligence will arise. And they would often say that the cognitive science people are trying to fully understand how intelligence works in order to reproduce it, which they clearly see as a vain pursuit.

But then, the cognitive science people would say that they are trying to get at the root mechanisms of intelligence, while the machine learning are just coming up with method to optimize a poor and shallow imitation of intelligence (which is evidenced by Hofstadter's term "genuine artificial intelligence", which is merely a smear on anyone not doing cognitive science... yes, scientists can be petty too).

I do have to say that I find myself leaning towards the machine learning camp, but I don't denigrate the work done in cognitive science... but the lack of measurable progress in decades makes it a bit disappointing, to say the least. Cognitive science often feels more like a discourse in philosophy than a practical or applied science, as very well evidenced by that article you linked to.