Spencer Lyon

# Random Notes

last update:

## Monte Carlo methods

This is part 2 in the reinforcement learning for economists notes. For part 1 see Reinforcement Learning Intro. For a list of all entries in the series go here Monte Carlo methods As defined in Reinforcement Learning Intro GPI encompasses the core ideas for all the RL algorithms in this book. One family of such algorithms is Monte Carlo (MC) methods. One distinguishing feature of these methods is that they can be implemented in a completely model-free way.

## Reinforcement Learning Intro

This is part 1 in the reinforcement learning for economists notes. For a list of all entries in the series go here These notes mostly follow Sutton, R. S., & Barto, A. G. (2015). Reinforcement Learning: An Introduction (2 Draft, Vol. 9). http://doi.org/10.1109/TNN.1998.712192 The notes themselves might be interpreted as a macro-economist’s view of reinforcement learning (RL). I will cast many objects/ideas from the book into the domain-specific framework I am familiar with.

## Temporal-Difference methods

This is part 3 in the reinforcement learning for economists notes. For part 1 see Reinforcement Learning Intro. For a list of all entries in the series go here Temporal Difference methods We continue our study of applying GPI to the RL problem by looking now at temporal difference (TD) methods. One step TD (TD(0)) Let’s begin our exploration of TD methods by considering the problem of evaluating or predicting the state-value function $V(s)$.

## Rust

openssl I couldn’t get any cargo projects that depend on open ssl to build. I just did: brew install openssl export OPENSSL_INCLUDE_DIR=/usr/local/opt/openssl/include export DEP_OPENSSL_INCLUDE=/usr/local/opt/openssl/include Answer came from here openblas I also had a hard time getting open-blas to build because it couldn’t find libgfortran I fixed it by running LIBRARY_PATH=/usr/local/Cellar/gcc/6.3.0_1/lib/gcc/6 cargo build --release

## GitHub pro tips

Post-commit hooks You can use github post commit hooks to send an HTTP payload to a server after every commit. The payload will contain data about the commit that you can then use to trigger arbitrary actions (e.g. run scripts) on the server. I’ve used a simple go library webhook to do this. To get it up and running I did the following: Install webhook with: go get github.

## Probability distributions

Some probability distributions that are useful (usually to economists) for one reason or another. Pareto Distribution The Pareto distribution has a simple CDF: $G(x) = 1 - \underline{x}^{\gamma}x^{-\gamma}$, where $\underline{x}$ satisfies $G(\underline{x}) = 0$ and $\gamma$ governs the variance. A useful property of the Pareto distribution is that when it is truncated, the truncated CDF is the same as the original, except that $\underline{x}$ is moved up.

## OSX tips

When I install Iterm fresh I need to do a few things in settings -> profiles -> keys: Change so left option is Esc+ Map option right arrow to escape sequence f Map option left arrow to escape sequence b I got the last two tips from here

## Notes on Deterministic Difference Equations

Deterministic Difference Equations Scalar First-Order Linear Equations The basic scalar first-order difference equation can be represented by: $$x_{t+1} = b x_t + c z_t, \quad t \geq 0$$ where $x_t, b, c, z_t$ are all real numbers. Since these equations are deterministic then we already know the sequence ${ z_t }$ and will assume it is bounded. If $z_t$ is constant for all $t$ then this equation is called autonomous.

## Using Docker

Common commands I use: boot2docker up: This launches the boot2docker daemon on osx. After running I then have to copy/paste the export statements printed by this command to set up ports. An alternative is $(boot2docker shellinit), which will do the copy/pase of exports for me. docker ps -a: lists all containers docker rm$(docker ps -a -q): remove all containers (running or not) docker images: list local images docker run IMAGE_NAME: runs the docker image.