Markov Chain: Definition and Representation with Solved Problems

A Markov Process is said to be Markov Chain if it assumes values from Discrete State Space (E).
A Markov Chain is a Discrete Time Markov Process for which the future behavior depends on the current/Present state only not the past states. This property is also called the memoryless property of Markov Chain or Markovian Property.

Jump to Probability and Queuing Theory Index Page

A Discrete Parameter Markov Chain (MC) is represented as {X_n,n ≥ 0}.
P(X_n+1 = j / X₀ = i₀, X₁ = i₁... ,X_n = i )
P(X_n+1 = j / X_n = i )
P_ij(n,n+1)
P_ij(1)

Representation of Discrete Parameter Markov Chain

The joint probability distribution of the Markov Chain is represented in two ways. They are:

1. Step Transitional Probability
2. Transition Probability Matrix

Step Transitional Probability

P(X₀ = i₀, X₁ = i₁, X_n-1 = i_n-1, X_n = i_n)
= P(X_n = i_n /X₀ = i₀, X₁ = i₁, X_n-1 = i_n-1).P(X₀ = i₀, X₁ = i₁, X_n-1 = i_n-1)
= P(X_n = i_n / X_n-1 = i_n-1).P(X₀ = i₀, X₁ = i₁, X_n-1 = i_n-1)
= P_in-1.in(1).P(X₀ = i₀, X₁ = i₁, X_n-1 = i_n-1)
= P_in-1.in(1).P(X_n-1 = i_n-1 / X₀ = i₀, X₁ = i₁,....., X_n-2 = i_n-2 ).P(X₀ = i₀, X₁ = i₁,....., X_n-2)
= P_in-1.in(1).P_in-2.in-1(1).P(X₀ = i₀, X₁ = i₁,....., X_n-2)
= P_in-1.in(1).P_in-2.in-1(1).P_in-3.in-2(1).......P_i0.i0(1).P(X₀ = i₀)

where P(X₀ = i₀) is an initial probability vector.

Transition Probability Matrix

Consider a Markov Chain {X_n,n ≥ 0} with discrete state space {0,1,2,3,......}, Then the Markov Chain can be determined by Transition Proability matrix as follows:

---

Solved Problems of Markov Chain

Solved Problem 1: Consider the sample weather model in which there are two states rainy(denoted by 0) and sunny (denoted by 1). The Transition probability Matrix is given by

P	=	(	0.5	0.5	)
			0.1	0.9

If the initial probability vector is P(0) = (1   0), then what is the probability that the third day will be sunny?

=Solution:

---

Solved Problem 2: The transition Probability Matrix of a Markov Chain having three states 1,2 and 3 is

and the initial Probability distribution is P(0) = (0.5  0.3   0.2). Find the following

i) P(X₂ = 2)
ii) P(X₃ = 3, X₂ = 2, X₁ = 1, X₀ = 3)
=

ii. P(X₃ = 3, X₂ = 2, X₁ = 1, X₀ = 3)
 = P(X₃ = 3 / X₂ = 2, X₁ = 1, X₀ = 3).P(X₂ = 2, X₁ = 1, X₀ = 3)
 = P(X₃ = 3 / X₂ = 2).P(X₂ = 2, X₁ = 1, X₀ = 3)   [Using Markovian Property]
 = P₂₃(1).P(X₂ = 2 / X₁ = 1, X₀ = 3).P(X₁ = 1, X₀ = 3)
 = P₂₃(1).P(X₂ = 2 / X₁ = 1).P(X₁ = 1, X₀ = 3)     [Using Markovian Property]
 = P₂₃(1).P₁₂(1).P(X₁ = 1 / X₀ = 3).P(X₀ = 3)
 = P₂₃(1).P₁₂(1).P₃₁(1).P(X₀ = 3)  
 = 0.3 * 0.3 * 0.4 * 0.2
 = 0.0072

---

Related Posts:

• Stochastic Process- Definition, Specifications, and Classification
• Stationary Distribution | Steady State Distribution with Solved Problems
• n-Step Transition Probability Matrix of a Two-state Markov Chain with Solved problems
• Classification Of States and Periodicity of Markov Chain with Solved Problems
• Birth Death Process
• Chapman-Kolmogorov Equation