# Perceptron

I’m learning about various machine learning algorithms, so I want to get recorded what I have learned and where I learned it, in part so that I can relearn it again after I inevitably forget it!

The perceptron is “baby’s first neural network.” It can be used successfully for learning binary classification of data that is linearly separable. The basic idea is that you have some training data that comes to you as vectors. You can start by guessing a weighting for those vectors, which is basically a guess at the subspace that separates your data (the weighting gives the normal vector for the subspace), or you can just initialize the weights to 0. Then you look at a random data point. First, you have to see how your current perceptron categorizes the data point, which you can do by taking a dot product of the weight vector with the data point vector and just looking at its sign.

If it is incorrectly classified, you need to adjust your weighting, which you do by subtracting (a scaling of) the vector of your current data point from your weighting vector, giving that normal vector a bit of a bump so that you will be correctly classifying the current data point. Then you pick another point and do the whole thing again. You are continuously adjusting your weights, so presumably your perceptron is getting better all the time. It is also useful to note that you need to use some kind of activation function to distinguish between correctly and incorrectly classified data points and it seems pretty typical to use a threshold step function.

Does this process ever end? Yes, it will provided that your data is linearly separable. You can even find the proof here. How long does it really take? I don’t know. Presumably it’s not the worst thing to do since lots of people reference it. What if your data isn’t really linearly separable? Well, it will go on forever, so you better pick a maximum number of iterations. Will it give you something reasonable after a reasonable number of iterations if you data is linearly separable-ish? No idea, but it seems like it might.

I read several useful pieces to figure out what I do know. I found this material from a presentation in graduate course on machine learning (there’s a lot of other interesting stuff in the webpage for the 2007 course http://aass.oru.se/~lilien/ml/). I also relied heavily on this material on perceptron from a CMU course in computer vision. Both of these sources have useful illustrations that I decided not to replicate here, so you should go look at them.The wikipedia page on perceptron had some good material, and I got curious a little about history, so I read http://web.csulb.edu/~cwallis/artificialn/History.htm.

# #9: Park on a Hill

When I was writing my dissertation I was part of the PhinisheD community, and there I learned the concept of “parking on a hill.” The idea is to not end your day’s work with the completion of a task, but rather to end it with a task started, ready to move on the next day. Much in the way that gravity will help you get started if you park a car in a hill, the gravity of having a task in progress can help you get started when you park a project on a hill.

This is a technique I could use more of in my life right now, so I am going to record some of my thoughts on this technique here. Right now I am working on a paper that I think I’ve been writing for three years. I’m very slow. This is a “back burner” project which means I often go for days or weeks without touching it. So if I can be sure to have it set up with something that will move me forward whenever I leave it then I think two things will happen. First, part of my brain will already be engaged in the next idea or task, which may push me toward getting back to the project and may help good ideas to percolate in my brain. Second, if I have jotted down something that will help me get started I may find it easier to remember and reconnect with the project the next time I sit down.

Here are some ideas for parking on a hill:

• Write half a sentence to give yourself a jumping off point. This can produce some anxiety — you probably have an idea of how you are going to finish the sentence, but what if you never get that idea back? But the fact is that leaving openness allows something new to come into your work. If you over-plan your next steps you may miss the exciting accidents and new ideas that come to you when you least expect it.
• Briefly outline the thing you want to do or write. Don’t over-plan since that won’t produce energy or excitement, but do be sure you have a next job that is easy and fun to dive into.
• Ask a question. For instance, I am working on a developing some materials for a class this fall, so I might end my work day with this question, “How can I refresh student’s understanding of percentages without talking down to my students and ignoring the understanding they already have?” Then I can start my day with some freewriting in response to that question, which may help me set up the activities I want to develop of the class.

# #8: Don’t choose, just do and notice

Lately, I have been meditating, and I’m up to six minutes a day which is OK. In meditation the point is to be present to the now, to observe and only to observe. There are many different techniques to carry this out and to some extent it doesn’t matter what technique you choose. You could do anything. you could practice by wiggling your toes and observing that. But it is useful to have a particular practice; it gives you a focus and a way to discipline your brain toward mindfulness. Whatever particular method you have chosen, you simply keep coming back to the present. If your mind strays, come back to your breath or your mantra, or whatever. Keep coming back.,

I’d love to find such a simple practice for all of life. But life isn’t so easy because life isn’t about watching, it’s about action. In meditation, on the other hand,the point is not to act but to observe. But life is doing, about actions. One of our actions can be sitting down to meditate (or engaging in a meditation practice in a way that does not involve sitting), but for most of us it would be a poor life if that was the only action we took. We have to take life-sustaining actions like making and eating food, caring for our immediate environment, and caring for our bodies. We take actions to care for others, like children. We take actions to connect with friends and family. We take actions to bring in money so that we can sustain a roof over our heads, food on the table, and have discretionary income. life is full of actions. To some extent, life is action.

Personally that’s where I run into trouble. The actions available to us are nearly limitless. I could write this right now, I could get up and eat, I could go to the bathroom, I could write an email that I want to send, I could work on a research project, I could clean the house, I could take a shower, I buy clothes, I could watch a movie, I could make food, I could garden, I could volunteer, I could plan the work I am going to do this summer, I could buy groceries, I could play bass. I could keep this list going for a long long time, and these are just actions that are available to me right at this minute.

I always get stuck in picking an action. Most actions are not essential, they don’t absolutely have to be done. Occasionally I have a task that absolutely must be done at a particular time, and I both hate and love those. I have to get my kids from school at 2:55 even if I might want to stay home and work on something else. I have to teach class as 9:30 even though I might want to spend time with my colleagues at work. Of course, I could choose not to pick up my kids and not to teach class, but the consequences of those choices are unpleasant enough that I don’t have to make a decision — I just do the thing that needs to be done. But I spend most of my day in a state of choice. Do I want to do this or that? And then I get stuck. How do I choose between this and that? Sometimes, like this morning, I have something pressing enough that I have to do it. In other words, I have put it off for a long enough time that the choice is now made for me. But when I have a choice, I am stuck. Having the choice produces a great deal of anxiety, because I worry about making the wrong choice.

To reduce my own anxiety and be more satisfied, then I need to remove the choice, or at least find a way to not find myself in that choice state very often. Choosing is uncomfortable for me. I have sometimes done this using randomness, particularly using using Mark Forster’s random method (I will probably write more about this sometime), I would love to find a way to use the meditation trick of always bringing myself back to one particular focus, as well as connecting to the idea of observation.

So here is my idea. It doesn’t really matter what actions I take. That’s not entirely true of course since there are bad actions I can take, but let’s assume I’m choosing between reasonably good possible actions. I can choose to do this or to do that, much like I can choose any of a variety of meditation methods. But like with meditation, it is important to have a some particular discipline to practice. What is that discipline that I can bring yourself back to repeatedly? I can keep coming back to a focus on what I am doing right now.

So during my day, I am going to work on letting go of choosing the right action. I am going to remind myself that the choice doesn’t really matter in the long run. Rather than choosing what to do next, I am going to simply pay attention to my action at a particular moment. I’m going to do, and to notice what is happening. The noticing shouldn’t be about judging or weighing whether I’m doing the right thing because that’s not the point. The point is simply to be present.

It’s not very exciting as a system because there’s no lists or reward system, but I’ll see how it goes. And of course, I’m working on another system, inspired by software development, that does involve lists.

# #7: Timer tricks

Procrastinators have long known that setting a timer can help bust through tasks. Whether it’s FlyLady setting a timer for 15 minutes to throw away trash, Merlin Mann advocated doing a “dash” to beat procrastination, the two-minute rule of David Allen’s GTD (Getting Things Done), or writers “sprinting” during NaNoWriMo, using a timer gives lots of us a push to really get things done.

The current biggest fad in timer-based productivity is the Pomodoro technique, named after a tomato timer apparently. I’ve used the pomodoro technique off and on. The essence of the pomodoro technique is to use a 25 minute timer for a stretch of work, followed by a mandatory short break (3-5 minutes). After four pomodors you take a longer break. You also track the pomodoros you spend on tasks, potentially helping you get better at estimating your time. There is also a discipline to using the pomodoro technique because you are not allowed to split or pause a pomodoro. If a distraction comes up you write it down and keep going on your original task. If you stop to address something, you have to start the pomodoro again and it doesn’t “count.”

The pomodoro technique isn’t my favorite because I don’t like the fixed time of 25 minutes, both because it isn’t a nice chunk of time (it is a very awkward fraction of an hour — I prefer 5, 10, 15, 20, or 30 minute chunks) and because I find I really need to vary the time that I use depending on my mood and the task at hand. If I really am dreading a task I need a 5 minute timer. I’ll never do it if I have to spend 25 minutes. On the other hand, if I am really into a task there is no way that I am taking that break. It doesn’t really work that well for me because I find I need something with variable times.

I have, however, used a lot of pomodoro timers. Most of them let you set variable times instead having to stick with 25 minutes. The best ones keep track of the number of pomodoros you spend on your various tasks. Pomodairo and Focus Booster are both pretty good ones that I have used. While I love the idea of tracking the time I spend on various projects and tasks, I’ve never been really happy with any of the ways I have found to do the tracking. These days if I need a timer, I tend to use 30/30, which gives me lots of flexibility to set up different repeating timers and routines.

# #6: A Spoonful of Music

It was just after I turned 30 that I started making playlists that got me out of bed and moving. I just finished recreating one that I used for a long time on Spotify (and you will simply have to forgive the fake Beatles since you can’t get real Beatles on Spotify). I made a number of these playlists and would typically associate each song with specific actions, like getting out of bed, eating breakfast, taking a shower, or brushing teeth. I also made specific playlists for things like working on my dissertation or doing household chores.

When my daughter was maybe 2 or 3 I wanted to do the same thing for her and found Childrens Miracle Music, which was the same thing only for kids. It had a little more religion than I would like, but it turned chores into the game, and in fact became known as “The Game” in our house. The game came with a chore chart and a happy voice telling everyone what to do next and how many songs we had to complete the next task. Beating the music was great motivation and the music kept the atmosphere fun. We used the game for a while, then stopped using it, and then later started again and I added my own voiced instructions and new songs. Eventually we decided to stop the game and move onto other systems, largely because the game required all of us to work at the same pace on the same chores, but I wouldn’t be surprised if chore playlists pop up again in our house in the future.