You must be familiar with recursion with normal functions, i.e a function calling itself. An anonymous function is a function without a name. So to use recursion in an anonymous function you call the arguments.callee() to refer to the anonymous function.

How to approach recursion?

The key idea to recursion is the terminating condition. First you need to figure out when the function should terminate. Then you think on how to generalize. Means first you look at when the factorial function say fact(n) should terminate, i.e, when n==1.

A normal function that computes factorial (I'm choosing factorial because its simple, and easy to understand) can be like:

function fact(n):int {
    if(n==1) return 1;
    else return n * fact(n-1);
}
trace(fact(5));

In terms of anonymous function:

var fact:int = (function(n):int {
    if(n==1) return 1;
    else return n * arguments.callee(n-1);
})(5);
 
trace(fact);