#stochastic process

This is my first ever /actual/ Tumblr post, so first of all, let’s hope I’m doing this right.

I’ll make my introduction fairly brief as to get to the mathematics ASAP. This was part of my senior project for my mathematics B.S. (which I am very close to completing!). I presented parts of this research at the Michigan MAA Section Meeting in April of 2017. And I am really excited to share what I found in a more casual setting (rather than in a formal paper for my class or a conference presentation).

So let’s get to it, eh?

For the first installment of this series, I want to touch on two main things:

Why Did I Decide To Do This?

Things You Need To Know Before We Can Get To The Cool Stuff (I Know, I Know, But You Need To Know These Things)

Why Did I Decide To Do This?

My senior mathematics capstone course was on the topic of Markov chains. The main purpose of this class was for each student to complete an individual research project. I struggled for quite some time to determine what I wanted to do my project on. I eventually settled on disease modeling, but found myself disinterested with the material. I credit one visit to my advisor’s office for what sparked in me the most academic drive and passion for a project I’ve ever had.

I walked in, planning on just talking about my ideas for my disease modeling project. My advisor suggested that I could use this sort of model to work with other things, and me, being the Internet-addicted smart ass that I am, asked, “Like memes?”

So here we are.

Things You Need To Know Before We Can Get To The Cool Stuff

First, I’m assuming that you are familiar with the following concepts from linear algebra:

Matrix algebra

Eigenvalues and eigenvectors

Diagonalizing matrices

For the sake of time and length, I leave reviewing those as an exercise to the reader. Ha, I’ve always wanted to say that.

So I’ll begin with Markov chains.

A Markov chain is a stochastic process where the probability of being in the current state is dependent solely on the previous state you were in.

What’s an example of this?

Suppose you are playing a game on a rectangular board of 7 squares. Your piece starts on the center square. For each move, you have a probability of 1/3 of moving to the left (which I'll denote L) and a probability of 2/3 of moving to the right (which I'll denote R). If you get all the way to the right, you win! However, if you get all the way to the left, you lose (*sadface*). Once you reach either the leftmost square or the rightmost square, you can’t move out of them. We’ll talk about this in more detail later, but these are what we call absorbing states.

See below for the starting position.

Say your first four moves are R, L, R, R. 

Your probability of moving to the right is still 2/3 and your probability of moving to the left is still 1/3. Much like the honey badger, the probabilities don’t care that you moved R, L, R, R already; they are still the same as before. Examining this as a Markov chain is nice because it allows us to answer two important and interesting questions: 

What is the probability of winning after a certain number of moves?

What is the average number of moves it takes until I win? 

I’ll attempt to answer question 1, and I’ll leave question 2 as an exercise.

In order to find the probability, we must first create a transition matrix. The transition matrix, $T$, is a matrix where $T_{i,j}$ is the probability of getting to the $i^{th}$ state from the $j^{th}$ state. In this example, each state is a square on our board. Our transition matrix looks something like this:

Let’s discuss!

We note that every column sums to 1. This makes sense! We have to move every turn, so the sum of the different probabilities of leaving each state should equate to exactly 1. We also note that $T_{1,1}$ and $T_{7,7}$ are both equal to 1. These are our absorbing states! Absorbing states are those states where the probability of returning to itself is exactly 1. So, you’re essentially trapped (mwa-ha-ha) in these absorbing states. 

How does this help us find the probability of winning?

Excellent question! If we want to find the probability of being in the $i^{th}$ state after starting in the $j^{th}$ state after n moves, we simply find $T^n_{i,j}$. This can be done using your favorite computing software (I personally opt for SageMath), or if you’re feeling particularly ambitious, you can do it by hand (but I don’t recommend this, for the sake of your health). 

So if I want to find the probability of winning the game in at most, say, 10 moves, I would look at $T^{10}_{7,4}$, as this entry shows the probability of ending in the 7th state (win!) from the 4th state. Additionally, I could also find the probability of losing after at most 10 moves by looking at $T^{10}_{1,4}$. 

Here’s $T^{10}$ as outputted by SageMath:

Our $T_{7,4}$ entry is $\approx$ 0.7133, giving us a nice 71.33% chance of winning the game in at most 10 moves!

We can continue to raise $T$ to higher and higher powers and see if we can find the values that the probabilities approach, but there’s an easier way! We can find the steady state distribution of our transition matrix. 

The steady state distribution is the vector that when we multiply $T$ by this vector, we just obtain the same vector. In other words, it’s a vector $v$ of probabilities such that $vT = v$. Also, this vector $v$ is actually the eigenvector associated with the eigenvalue of 1!

For the sake of time, and since I’m assuming you’re comfortable with matrix algebra, I’ll leave the solution to you.

After finding the steady state distribution, we can determine that the overall probability of winning is 8/9 and the overall probability of losing is 1/9. Good odds! 

How does this fit into my research? / Preview for the next post!

I’ll talk more in the next post about how I viewed the spread of memes as a Markov chain, but in general, I worked sort of backwards from how I solved this example. In my research, I don’t know my probabilities, and my goal is to try to find them! So I use these ideas to create some ~pretty dope~ equations involving these unknown probabilities and try to match these equations to data sets I obtain from Google Trends. Of course, this will make more sense with my future posts!

Hope you enjoyed! Any questions/comments/concerns, feel free to send me an ask. 

Best wishes and stay positive!

(I promise that wasn’t a math pun)

(Okay, maybe it was)

In probability theory, a Markov model is a stochastic model used to model randomly changing systems.

Stochastic process

Let’s see what Stochastic process is first .

Stochastic is, having a random probability distribution or pattern that may be analysed statistically but may not be predicted precisely., or in simple terms refers to a randomly determined process . In probability theory and related…

[latexpage] 我经常碰到一些貌似简单，实则复杂的问题。因为貌似简单，以为两下能搞定，一开始就陷了进来；后来发现实际很复杂，但却浪费了大量的时间。详见我在知乎上问的问题和答案。我的答案最后的思路计算量很大。现在还在请教数学专业的人，看有没有别的办法。无论如何，仍然很奇怪，一直在怀疑是不是我自己搞错了，因为一个随机行走的均方回转半径，听起来应该是一个非常常规的问题，哪怕可能是复杂的，但也应该早有人解决了，不致于连资料都查不到。

Aleksi Perälä Stochastic Process (2014)