The paper explores the application of reinforcement learning (RL) and multi-armed bandit (MAB) algorithms, primarily focusing on the epsilon-greedy strategy, for problems ranging from simulated slot machine optimization to identifying potentially rising stocks in the KOSPI market.

The core methodology begins with an introduction to Reinforcement Learning, defining an agent that interacts with an environment, selects actions, receives rewards, and learns an optimal policy to maximize cumulative rewards. The Multi-Armed Bandit (MAB) problem is presented as a fundamental RL scenario, where an agent must choose among several slot machines, each with a different, unknown reward probability, to maximize total reward over time. This introduces the critical dilemma of exploration (trying new, potentially better options) versus exploitation (choosing the currently known best option).

Two primary methods for estimating slot machine probabilities (Q-values) are discussed:

1. Experimental Probability Estimation (Sample Average): This involves repeatedly playing a slot machine and calculating the empirical average of rewards. For a sequence of rewards $R_1, R_2, \dots, R_n$, the estimated probability $Q_n = \frac{1}{n} \sum_{i=1}^{n} R_i$.
2. Recursive Update (Incremental Sample Average Update): This method updates the estimated Q-value incrementally with each new reward. If $Q_{n-1}$ is the estimated mean after $n-1$ trials and $R_n$ is the reward from the $n$-th trial, the new estimate $Q_n$ is updated as:

$$Q_n = Q_{n-1} + \frac{1}{n}(R_n - Q_{n-1})$$
This method is more computationally efficient as it does not require storing all past rewards. It is also noted to be a form of Exponential Moving Average (EMA) when considering a constant learning rate.

The paper then details the implementation of an epsilon-greedy agent for the MAB problem. A Game class simulates slot machines, each having a randomly assigned success rate. An Agent class embodies the epsilon-greedy policy:

• With probability $\epsilon$ (exploration), the agent chooses a random action (slot machine).
• With probability $1 - \epsilon$ (exploitation), the agent chooses the action (slot machine) that currently has the highest estimated Q-value, i.e., $a^* = \text{argmax}_a Q(a)$.

The agent's update method uses the recursive update rule mentioned above: $self.Qs[action] += (reward - self.Qs[action]) / self.Ns[action]$, where Ns[action] is the count of times action has been selected. The performance of this agent is evaluated over multiple runs and steps, calculating total_reward and rates (average reward per step). Comparisons are made by varying the $\epsilon$ value (e.g., 0.01, 0.1, 0.3) to demonstrate the trade-offs between exploration and exploitation.

A more advanced Q-learning algorithm with a constant learning rate (alpha-constant update) is introduced to handle environments where slot probabilities can dynamically change. A Game2 class is used, where self.rates (slot success probabilities) are perturbed with noise ($self.rates += 0.1 * np.random.randn(self.arms)$) after each play. An Agent2 class is introduced with a self.alpha (learning rate). Its update method implements the Q-learning update rule:
$$Q_t = Q_{t-1} + \alpha(R_t - Q_{t-1})$$
This formula updates the Q-value based on the difference between the observed reward $R_t$ and the current Q-value $Q_{t-1}$, scaled by the constant learning rate $\alpha$. This approach gives more weight to recent rewards, making it suitable for non-stationary environments. The paper compares the performance of the "sample average" agent and the "alpha constant update" agent.

Finally, the paper applies the Q-learning concept to identify potentially rising stocks in the KOSPI market. For each KOSPI stock, daily closing prices are analyzed. A reward is defined as 1 if the next day's closing price increases by 5% or more compared to today's, and 0 otherwise. A Q-value is maintained for each stock, representing its estimated "value" or potential for positive returns. The Q-value for each stock is updated daily using the Q-learning rule:
$$Q_{stock, t} = Q_{stock, t-1} + \alpha(R_{stock, t} - Q_{stock, t-1})$$
where $\alpha=0.8$. A specific condition, $if Q > 0.9$, is used to filter for high-potential stocks. Only when a stock's Q-value exceeds 0.9 (indicating a high expected reward) are its subsequent rewards considered for calculating a "reward rate" (count / total_count), which tracks the frequency of positive returns among high-Q stocks. Stocks with a final Q-value greater than 0.9 are then identified as "rising stocks."