The trace-reinforced ants process does not find shortest paths

In this paper, we study a probabilistic reinforcement-learning model for ants searching for the shortest path(s) between their nest and a source of food. In this model, the nest and the source of food are two distinguished nodes $N$ and $F$ in a finite graph $\mathcal G$. The ants perform a sequence of random walks on this graph, starting from the nest and stopped when first hitting the source of food. At each step of its random walk, the $n$-th ant chooses to cross a neighbouring edge with probability proportional to the number of preceding ants that crossed that edge at least once. We say that {\it the ants find the shortest path} if, almost surely as the number of ants grow to infinity, almost all the ants go from the nest to the source of food through one of the shortest paths, without loosing time on other edges of the graph. Our contribution is three-fold: (1) We prove that, if $\mathcal G$ is a tree rooted at $N$ whose leaves have been merged into node $F$, and with one edge between $N$ and $F$, then the ants indeed find the shortest path. (2) In contrast, we provide three examples of graphs on which the ants do not find the shortest path, suggesting that in this model and in most graphs, ants do not find the shortest path. (3) In all these cases, we show that the sequence of normalised edge-weights converge to a {\it deterministic} limit, despite a linear-reinforcement mechanism, and we conjecture that this is a general fact which is valid on all finite graphs. To prove these results, we use stochastic approximation methods, and in particular the ODE method. One difficulty comes from the fact that this method relies on understanding the behaviour at large times of the solution of a non-linear, multi-dimensional ODE.


Introduction and main results
Context.-It is believed that ants are able to find shortest paths between their nest and a source of food with no other means of communication than the pheromones they lay behind them.This phenomenon has been observed empirically in the biology literature (see, e.g., [GADP89,DAGP90,MTNS13]), and reinforcement-learning has been proposed as a model for describing it (see, e.g.[MTNS13]).In the survey of [DS04, Ch. 1], this phenomenon is called stigmergy: "ants stimulate other ants by modifying the environment via pheromone trail updating".
In this paper, we study a probabilistic reinforcement-learning model for this phenomenon of ants finding shortest path(s) between their nest and a source of food.In this model, which was introduced in our previous paper [KMS20], the nest and the source of food are two nodes N and F of a finite graph G , and the ants perform successive random walks starting from N and stopped when first hitting F .The distribution of the n-th ant's walk depends on the trajectory of the previous n − 1 random walks in a way that models ants leaving pheromones on the edges they cross: each edge has a weight which is equal to 1 at the start and which increases by 1 at time n if and only if the n-th ant has deposited pheromones on that edge.The transition probabilities of the random walk of the (n + 1)-th ant are proportional to the edge-weights at time n.The reinforcement is thus linear.
In [KMS20], the ants deposit pheromones only on their way back: i.e., when they come back to the nest after having hit F .Two cases are studied in [KMS20]: -In the "loop-erased" ant process, or model (LE), ants come back to the nest following the loop-erasure of the time-reversed version of their forward trajectory.
-In the "geodesic" ant process, or model (G), they come back following the shortest path between F and N within the trace of their forward trajectory (ties are broken uniformly at random).
The conjecture is that, under these two versions of the model, when time goes to infinity, almost all ants go from N to F through a shortest path, i.e., the ants indeed find the shortest path(s) between their nest and the food.This conjecture is proved in [KMS20] for model (LE) in the case when G is a series-parallel graph, and in the model (G) in the case when G is a five-edge graph called the lozenge.
Main contribution.-In this paper, we look at the same model but assuming that ants deposit pheromones on their way forwards to the food, i.e., the weight of an edge increases by one at time n if and only if the n-th ant has crossed this edge at least once on its way from N to F .We call this model the "trace-reinforced" ant process, or model (T).In the biology literature, all cases of ants depositing pheromones on their way forwards, backwards or both are considered (see [DS04,Ch. 1]).
Maybe surprisingly, this small change to the reinforcement rule leads to a drastically different behaviour: indeed, we prove that, in the trace-reinforced ant process, in general, the ants do not find the shortest path(s) between their nest and the source of food, except in some very particular cases.Indeed, in Theorem 1.3, we show that on a J.É.P. -M., 2022, tome 9 family of graphs called "tree-like", in which there is a unique edge between N and F (N and F are at distance 1), the ants do find the shortest path.However, one can find graphs, with N and F at distance one, which are not tree-like and such that the ants do not find shortest paths (see Proposition 1.4, Theorem 1.5 and Proposition 1.6).
The fact that ants do not always find shortest paths when depositing pheromones only on their way forward has been observed empirically on ants (see [DS04, §1.1.2]):"The observation of real ant colonies has confirmed that ants that deposit pheromone only when [going forward] to the nest are unable to find the shortest path between their nest and the food source".Our analysis proves that the model introduced in [KMS20] exhibits the same behaviour.
Our second main finding, which might also be surprising at first glance, is that although we use a linear reinforcement mechanism (see the discussion below), in all the graphs we consider, the sequence of normalised edge-weights converges to a deterministic limit.In fact, we conjecture that this is a general phenomenon, and that on all graphs with no multiple edges linked to node F , the sequence of the normalised weights converges almost surely to a deterministic limit: Conjecture 1.1.-If, for all n 0, for all e ∈ E, we let W e (n) denote the weight of edge e at time n, then there exists a deterministic family (χ e ) e∈E such that (W e (n)/n) e∈E → (χ e ) e∈E almost surely when n → +∞.Furthermore, if in addition the distance between N and F is at least 2, then χ e > 0 for all e ∈ E.
Discussion. -Other probabilistic reinforcement models inspired by urns exist in the probability literature.As far as we know these models are all self-reinforced random walks models with super-linear reinforcement; see for example Le Goff and Raimond [LGR18] and Erhard, Franco and Reis [EFR19].
The ant process can be seen as a "path formation" model: the quantity of interest is the subgraph of G obtained by removing from G all edges whose normalised weight converges to zero.If this limiting graph is different from G , then we say that "some path(s) has formed".
Another model for path formation is the Pólya urns with graph based interactions of Benaïm, Benjamini, Chen and Lima [BBCL15] and its generalisation to WARMs of [vdHHKR16].In the latter (and in more recent paper on the same model such as Hirsch, Holmes and Kleptsyn [HHK20]), only super-linear reinforcement is considered because it leads to path formation; in contrast, it is believed that, under linear reinforcement, the limiting graph would be equal to G .In [CH21], Couzinié and Hirsch consider the sub-linear WARM model.They show that, for bounded degree (possibly infinite graphs), if the reinforcement is sufficiently weak, or if G = Z, then the limiting edge-weights exist and are deterministic.In [BBCL15], and later [CL14] and [Lim16], the cases of sub-linear, super-linear and linear reinforcement are considered.In the linear case, if the original graph G is regular and not bipartite, then the vector of normalised weights converges almost surely to a non-deterministic limit.
Given these examples from the literature, it is quite surprising that in the ant process, with linear reinforcement, the vector of normalised weights always converges to a deterministic limit.In fact, this difference of behaviour can be observed when looking at the simplest probabilistic model with linear reinforcement: (generalised) Pólya urns.A d-colour Pólya urn of replacement matrix R = (R i,j ) 1 i,j d is defined as follows: at time zero, there is one ball of each colour in the urn, and at every time step, we pick a ball uniformly at random among the balls in the urn, and if its colour was i, we return it to the urn together with R i,j balls of colour j (∀j such that 1 j d).If R is the identity, it is known (see Markov [Mar17]) that the vector U (n) whose coordinates are the number of balls of each colour divided by n converges almost surely to a Dirichlet (1, . . ., 1)-distributed random variable.In contrast, if the matrix R is irreducible, then U (n) converges almost surely to a deterministic limit (see Janson [Jan04] and the references therein).
Interestingly, in a not-yet-available paper [HK] (whose results have been announced), Holmes and Klepstyn also exhibit a linear reinforcement model which, on some graphs, converges to a deterministic limit, a result that resonates with Conjecture 1.1.
A similarity between this paper, the Pólya urns with graph-based interaction of [BBCL15] and the WARMs of [vdHHKR16] is the method of proof since we also use stochastic approximation.In particular, we use the ODE method for stochastic approximation, and apply results from Benaïm [Ben99] and Pemantle [Pem07].However, the analysis of these stochastic approximations in the different examples of graphs we consider is quite different from the one in [BBCL15] and [vdHHKR16].In fact, as we later explain in more details, each of our examples requires an ad-hoc argument, which suggest that proving our conjecture on the convergence of edge-weights in great generality is a difficult open problem.
That said, we prove a general result ensuring that on any finite graph, the sequence of normalised edge weights is a stochastic approximation, in a sense which is recalled in Definition 2.3 below, with a vector field F which is Lipschitz on a suitable convex compact subspace of the Euclidean space (see Proposition 2.7).
1.1.Definition of the model and main results.-Let G = (V, E) be a finite (undirected) graph with vertex set V and edge set E, with two distinct marked nodes called N (for "nest") and F (for "food").We define a sequence (W (n) = (W e (n)) e∈E ) n 0 of random weights for the edges of G recursively as follows: -At time zero, all weights are equal to 1, i.e., W e (0) = 1 for all e ∈ E.
-Given W (n), we sample a random walk (X (n+1) i ) i 0 on G according to the following distribution: -the walk starts at node N , i.e., X (n+1) 0 = N , -it stops when first hitting F , i.e., P(X where u ∼ u if there is an edge linking the vertices u and u.
J.É.P. -M., 2022, tome 9 We let γ(n + 1) be the set of edges that were crossed at least once by the random walk X (n+1) ; we call this the "trace" of the (n + 1)-th walker.For all e ∈ E, we set W e (n + 1) = W e (n) + 1 e∈γ(n+1) .
We call this process the "trace-reinforced" ant process on G .In our first result, we focus on graphs that are "tree-like" in the following sense: Definition 1.2.-We say that a graph G = (V, E) with two marked nodes N and F is tree-like if the graph whose vertex set is V {F } and whose edge set is E minus all edges that contain F is a tree (i.e., a graph with no cycle).
Theorem 1.3.-Assume that G = (V, E) is tree-like and that the edge a = {N, F } belongs to E with multiplicity 1.Then, almost surely when n → +∞, In other words, following this reinforcement algorithm, the ants eventually find the shortest path between their nest and the source of food, i.e., the proportion of ants that go from N to F by only crossing the edge {N, F } is asymptotically equal to one, and the proportion of ants that cross any other edge asymptotically equals zero.
Note that if the edge {N, F } appears with multiplicity in E (i.e., there are edges from N to F in parallel), then it is easy to deduce from Theorem 1.3 that the normalised weights of all other edges go to zero almost surely, and the weights of the edges from N to F converge almost surely, as a -tuple, to a Dirichlet random variable with parameters (1, . . ., 1).
A natural extension of the set of tree-like graphs is the set of series-parallel graphs, which were considered for instance in [HJ04] and in our previous paper [KMS20].One could then ask whether the previous theorem extends to this class of graphs, that is, if the distance from the source to the food is one, do the weights of all edges not directly connected to both N and F go to zero?Maybe surprisingly, the answer is no.Indeed our next result provides a counter-example, which is depicted by Figure 1 and which we call the cone graph.
The following result shows that Theorem 1.3 does not extend to tree-like graphs where N and F are at (graph-)distance at least 2 from each other.For two integers p 1 and q 1, we define the (p, q)-path graph as the graph with two parallel paths between N and F , one of length p and one of length q (see Figure 2).
Theorem 1.5.-Let G = (V, E) be the (p, q)-path graph.We let a 1 , . . ., a p (resp.b 1 , . . ., b q ) denote the edges of the path of length p (resp.q), numbered from the closest to the nest to the closest to the food.
If min(p, q) 2, then, almost surely for all 1 k p and 1 q, where (α, β) is the unique solution in (0, 1) 2 of (1.1) Note that, if p = q 2, the solution of (1.1) is explicit and given by α = β = 2 −1/p .Now to give further support to our conjecture that normalised edge weights always converge to a deterministic limit (when the edges connected to F are simple), we look at the lozenge graph in Figure 3; this example, which was also considered in [KMS20], is different from all other cases so far, in the sense that it does not belong to the class of "series-parallel" graphs.
Proposition 1.6.-Let G be the lozenge graph of Figure 3.If we let W i (n) denote the weight of edge i at time n (with the edges numbered as in Figure 3), and W (n) = (W i (n)) 1 i 5 , then almost surely as n → +∞, we have where w * is the unique solution of 2x 3 + 4x 2 − 2x − 3/2 = 0 in (0, 1).
The fact that the limiting weights of edges 2 and 5 are equal to 1/2 should not come as a surprise.Indeed, by symmetry they must be equal (assuming they are deterministic), and since each ant reinforces exactly one of these two edges at each step, their common value has to be 1/2.Similarly for the edges 1 and 4, except that since each ant can reinforce both of them, one has now w * > 1/2.However, the fact that the limiting weight of edge 3 equals 1/2 seems to be merely a coincidence.
Notation.-Given some filtration (F n ) n 0 , and Z some random variable, we will use the notation E n [Z] to denote the conditional expectation of Z with respect to F n .
J.É.P. -M., 2022, tome 9 Acknowledgements.-We thank three anonymous referees for their careful reading of the paper and their constructive comments.Preliminary investigations on the problems treated in this paper were carried out by Yassine Hamdi, a student at École Polytechnique at the time, during an undergraduate research internship at the University of Bath, under the supervision of CM (see [Ham20] for Yassine's internship report).The authors are grateful to Yassine for the time he spent on these questions, and to both the École Polytechnique and the University of Bath for making this internship possible.

Preliminaries
2.1.Urn processes.-We state here a result concerning (generalised) Pólya urn processes.Given a function G : [0, 1] → [0, 1], we call G-urn process, a process (X n ) n 0 with integer values, such that almost surely X n+1 ∈ {X n , X n + 1}, and for all n 0, with X n := X n /(n + 2).In general we will assume that it starts from 1 at time 0, i.e., that X 0 = 1, but we shall also consider other initial conditions.We then say that it starts from some value k at time m, if we condition the process on the event {X m = k}.
Informally X n corresponds to the number of (say) red balls after n draws in a Pólya urn with two colours, where at each step, we draw a ball in the urn at random, and replace it into the urn with an additional ball of the same colour.At each draw, the probability to pick a red ball is G(p) if the proportion of red balls in the urn is p.
We will need the following standard result (which follows for instance from Corollary 2.7 and Theorem 2.9 in [Pem07]).
Proposition 2.1.-Let (X n ) n 0 be a G-urn process, with G a C 1 -function.Then almost surely ( X n ) n 0 converges towards a stable fixed point of G, that is a (possibly random) point p ∈ [0, 1], such that G(p) = p and G (p) 1.
In particular if there exists c > 0, such that G(x) > x, for all x ∈ (0, c) We shall also use the following corollary.
By assumption on G, one has G(x) G ε (x), for all x ∈ [0, 1].It follows that (X n ) n 0 stochastically dominates a G ε -urn process, and applying Proposition 2.1, we deduce that almost surely lim inf X n c − ε.Since this holds for all ε ∈ (0, c), the result follows.
In our applications, the process (X n ) n 0 will often be one coordinate of a higherdimensional process (X n ) n 0 , and (F n ) n 0 will simply be the natural filtration of the process (X n ) n 0 .
2.2.Stochastic approximation and the ODE method.-We use the following definition for a stochastic approximation (note that we do not seek for the most general definition here, but it will be sufficient for our purpose).
where the vector field F : E → R is some Lipschitz function, the noise ξ n+1 is F n+1 -measurable and satisfies E n [ξ n+1 ] = 0, for all n 0, and the remainder term r n is F n -measurable and satisfies almost surely r n C/n, for some deterministic constant C > 0.
Remark.-The fact that we assume E to be a convex compact subset of R d enables to easily extend F into a Lipschitz continuous function defined on R d , simply by composing it with the orthogonal projection on E .We then fall into the setting of Benaïm [Ben99], and we can rely on its results.Thus in the following, we will identify F with its Lipschitz extension on R d , as defined here.
Remark.-The choice of renormalisation factor equal to 1/(n + 2) in Equation (2.2) is arbitrary and can be replaced by 1/(n + 3) (as in Sections 4 and 6).In Sections 4 and 6, we explain why the chosen renormalisation is the most convenient.
The idea underlying the ODE method is that the trajectories of a stochastic approximation asymptotically follow the solutions of the differential equation We recall that if for x ∈ R d , we let (Φ t (x)) t 0 be the (unique because F is Lipschitz) solution of (2.3) starting at x, then this defines a flow, in the following sense.
Definition 2.4.-Let M be some metric space.A flow (or semi-flow) on M is an application Φ : R + × M → M , such that Φ 0 = Id, and Φ t+s (x) = Φ t • Φ s (x), for all s, t 0, and x ∈ M .
-A subset A ⊂ M is said to be invariant, if Φ t (x) ∈ A , for all x ∈ A and all t 0.
Theorem 2.5.-Let (X n ) n 1 be a stochastic approximation.If there exists a deterministic constant C > 0, such that almost surely sup n 1 ξ n C, then almost surely, the limiting set L(X) = n 0 m n {X m } is invariant by the flow of the ODE (2.3), connected, and the flow of the ODE restricted to L(X) admits no other attractor than L(X) itself.
We will also use the following corollary of Theorem 2.5: Corollary 2.6.-Under the assumptions of Theorem 2.5, if there exist a set U ⊆ E and p ∈ U such that (i) almost surely, L(X) ⊆ U , and (ii) for all w ∈ U , the solution of the ODE (2.3) started at w converges to p, then L(X) = {p} almost surely.
Proof.-First note that either L(X) = {p} or there exists a (possibly random) point x ∈ L(X) {p}.In the first case, the conclusion of Corollary 2.6 holds trivially.In the second case, we show that (i) and (ii) together with Theorem 2.5 imply that p ∈ L(X).Indeed, by (i), we have that x ∈ U .Thus, by (ii), the solution t → Φ(t) of the ODE started at x converges to p when time goes to infinity.Theorem 2.5 ensures that L(X) is invariant by the flow of the ODE, and thus that Φ(t) ∈ L(X) for all t 0. Since L(X) is closed (as the intersection of closed sets), this implies that lim t→+∞ Φ(t) = p ∈ L(X), as claimed.Finally, by Theorem 2.5, the flow restricted to L(X) admits no other attractor than L(X) itself, and by (ii), p is an attractor of the ODE restricted to L(X) ⊆ U ; this implies that L(X) = {p}, as required.

2.3.
The process of edge weights seen as a stochastic approximation.-Consider a finite graph G = (V, E), with two marked vertices N and F .Recall that W (n) = (W e (n)) e∈E denotes the sequence of the weights of the edges of the graph after n steps of the trace-reinforced ant process, and let For any edge e ∈ E, and any n 0, we let X e (n) := W e (n)/(n + 1), and X(n) = (X e (n)) e∈E .Next for any w ∈ [0, 1] E , and any e ∈ E, we let p e (w) be the probability that the edge e belongs to the trace of a random walk on the graph G endowed with the weights w, starting from N and killed at F .Then we define F : F e (w) := p e (w) − w e , for any e ∈ E.
J.É.P. -M., 2022, tome 9 Given w ∈ [0, 1] E , we set where e ∼ x means that we sum over all edges e ∈ E that have x ∈ V as endpoint, and recall that this defines a reversible measure for the random walk on G endowed with the weights w.We also let S(G ) be the number of self-avoiding paths from N to F in G , which we number in some arbitrary order: c 1 , . . ., c S(G ) .For i = 1, . . ., S(G ), we define and w e 1/S(G ) for all e ∈ c i .
Note that each E i is a convex compact subset of [0, 1] E .Then we further define, (2.5) i=1 λ i w i : i λ i = 1, and λ i 0, for all i w i ∈ E i , for all i , the convex hull of the union of the E i 's, which is also a convex compact subset of [0, 1] E .One has the following general fact.
Remark.-We stress that the proof of this result works in a wider setting, including the two variants of the process considered in our previous paper [KMS20].
Proof.-For the first part, we use a coupling argument.For all w, w ∈ E , we define (X n ) n 0 as the random walk on G equipped with edge-weights w, and (X n ) n 0 as the random walk on G equipped with edge-weights w .Both walks start at N and are killed when they first reach F .We couple (X n ) n 0 and (X n ) n 0 until the first time when they differ, in a way that maximises the probability that they stay equal after each step.We let τ be the random time when the walks first differ.If τ F denotes the first time when (X n ) n 0 hits F , then (2.7) where the first inequality is obtained from (2.4) using the triangle inequality, the second inequality is a general fact which holds for any coupling of the two walks, since if an edge is crossed by only one of the two walks, then necessarily τ τ F , and the last equality in (2.7) holds because our coupling maximises the probability of the two walks staying equal.From (2.7), we get with G w (•, •) the Green's function on the graph G endowed with the weights w (i.e., the mean number of visits to the second argument for a random walk starting from the first argument, up to its hitting time of F ). Using the reversibility of the measure π w , we deduce that (see e.g.[LP16, Ex. 2.1(e)]), Using also that G w (x, N ) G w (N, N ), we get with w − w 1 = e∈E |w e − w e |.Now by definition, for w ∈ E , one has π w (N ) 1, and we claim that G w (N, N ) is also bounded by a positive constant independent of w (only depending on the graph G ). Indeed, by [LP16, Eq. (2.4)], for a random walk starting from N , the number of returns to N before hitting F is a geometric random variable with mean π w (N )/C (G ,w) (N, F ), where C (G ,w) (N, F ) denotes the effective conductance between N and F in the graph G endowed with the weights w.
Moreover, by definition of E , for any w ∈ E , there exists a self-avoiding path from N to F such that all the edges on this path have a weight larger than S(G ) −2 (we recall that S(G ) denotes the number of self-avoiding paths between N and F in G ).Such a path has an effective conductance larger than denotes the maximal length of a self-avoiding path from N to F in G .By Rayleigh's monotonicity principle, we also have Finally, note that π w (N ) is bounded by the degree of N , say d G (N ).In total, this implies that, for all w, w ∈ E , a constant which only depends on the graph G .This concludes the proof of the fact that F is Lipschitz on E .J.É.P. -M., 2022, tome 9 Since (2.6) is straightforward by definition of the model, it only remains to show that X(n) belongs to E , for all n 0. The fact that π X(n) (N ) 1 follows from the fact that, by definition of the model, at each step at least one of the edges incident to N is reinforced.Furthermore, at each step, at least one of the self-avoiding paths from N to F is reinforced, which implies that, at any time n 0, at least one of these self-avoiding paths has been reinforced at least n/S(G ) times.In other words, for all n 0, X(n) belongs to at least one of the E i 's, and thus X(n) belongs to E as claimed.
2.4.Case when N and F are at distance one.-In this section, we prove the following general fact: if N and F are at distance 1, then the only simple paths from N to F which "survive" asymptotically are those of length one.In other words, asymptotically, almost all of the ants reach F by last crossing one of the edges that connect N and F .However, it is not true in general that the only edges which survive are those from N to F ; the cone graph of Proposition 1.4 is a counter-example.
Proposition 2.8.-Assume that G is a finite graph with two marked vertices N and F , connected by at least one edge.Let (W (n)) n 0 be the process of edge-weights of the trace-reinforced ant-process on G .Then for any edge e connected to F but not to N , one has W e (n)/n → 0 almost surely.
Proof.-First note that it is enough to prove the result in the case when there is a unique edge a = {N, F } (i.e., it has multiplicity one).Indeed, if {N, F } has multiplicity m 2, then, by definition of the ant process, at most one of these m edges belongs to the trace of each ant.Hence, the process obtained by adding the weights of theses edges into one weight is the ants process on the graph in which the m parallel edges have been merged into one edge with initial weight m.
Assume that w is such that w a / ∈ {0, 1}.Let G be the graph obtained by removing edge a from G : i.e., G = (V, E ) where E = E {a}.We equip the edges of G with the weights (w e ) e∈E , and let C G (w) denote the conductance between N and F in G equipped with these weights.We denote p a (w) the probability to reinforce the edge a when the weights over the graph are given by w.With this notation, we have .
We let k denote the number of edges connected to N in G , and denote the number of edges connected to F in G .We define the graph G (k, ) as the graph with vertex set {N, F, P } and with edge set {N, P } with multiplicity k and {P, F } with multiplicity (see Figure 4).We equip the k edges between N and P in G (k, ) with the same weights as the k edges connected to N in G , and the edges between P and F in G (k, ) with the same weights as the edges connected to F in G .The graph G (k, ) can be obtained from G by merging all vertices different from N and F into one node called P , or equivalently by adding edges between all pair of vertices disjoint of N and F , and assigning an infinite weight to all edges not connected to N or F .The conductance of G (k, ) with these weights is given by where X ∼ e denotes that the vertex X is an endpoint of e, where we have used that for all e ∈ E , w e 1, and also that e∈E : F ∼e w e = 1, and thus Therefore, p a (w) w a w a + k(1−wa) k+1−wa .
Thus, (W a (n)) n 0 stochastically dominates a G-urn process with Note that G(x) = x if and only if x ∈ {0, 1}, and one can compute that Thus Corollary 2.2 shows that W a (n)/n converges almost surely to 1, and as a consequence one also has that W e (n)/n converges almost surely to 0 for all e connected to F different from a.This concludes the proof.

Proof of Theorem 1.3
First note that Proposition 2.8 implies that W a (n)/n → 1, almost surely.Note also that by definition of the model, it now suffices to show that the normalised weight of all the other edges connected to N go to zero almost surely, as the weight of any edge e in the tree is always smaller than the weight of the unique edge connected to N on the path from e to N .
So let e be some edge connected to N , which is different from a.By assumption, the graph G is a tree rooted at N and whose leaves have been merged into F .Thus, since the ants are stopped when first hitting F , the first time each ant crosses the edge e has to be from N to the other extremity of e.Moreover, if an ant crosses the edge a it is stopped immediately.This implies that the probability to cross e before reaching F on G is smaller than on the graph consisting only of the two edges a and e (in parallel between N and F ).This gives for all n 0, almost surely, , where we recall that we write W (n) = W (n)/(n+2).Now let ε > 0 be fixed.We know that almost surely, for n large enough, W a (n) 1 − ε.On the other hand on the event when W a (n) 1 − ε, for all n larger than some integer n 0 , we know by (3.1) that (W e (n)) n n0 is dominated by a G-urn process with G(x) = x/(x + 1 − ε).Then Proposition 2.1 shows that almost surely, lim sup n→∞ W e (n) ε.Since this holds for all ε > 0, this concludes the proof.

The cone
We first note that by Proposition 2.7 and the specific features of the cone, the process W (n) := W (n)/(n + 2) is a stochastic approximation on the space with E as defined in (2.5).More precisely, for all n 0, we have with ξ n+1 some martingale difference, and for all 1 i 4, F i (w) = p i (w) − w i , where p i (w) is the probability that edge i belongs to the trace of a random walk on the graph endowed with weights w, which starts from N and is killed at F .
Remark.-Note that, for the cone graph, it is convenient to define W (n) as W (n)/(n + 2) rather than as W (n)/(n + 1).This is because, by definition of the process, for all n 0, W 1 (n) + W 4 (n) = n + 2; indeed, W 1 (0) + W 4 (0) = 2 and, at each step we increase exactly one of W 1 (n) and W 4 (n) by one.
To calculate p 2 (and thus p 3 , by symmetry), we decompose according to the first step of the ant: to reinforce edge 2, it has to either go straight through edge 2 (in which case, no matter what it does later, edge 2 will be reinforced), or go through edge 3.In the latter case, the second step of the ant has to be either through edge 2 (edge 2 gets reinforced no matter what happens next) or back through edge 3, in which case we start again.Hence, Solving this in terms of p 2 (w) gives To calculate p 1 (w), we use the electrical networks method (see, e.g.[LL10]) to see that the probability, starting from N , to cross edge 1 before edge 4 is w 1 divided by the total conductance between N and F .The latter conductance is given by w 1 + (w2+w3)w4 w2+w3+w4 , which yields Thus, the coordinates of the function F are given by Now our aim is to use the ODE method and for this we proceed in 4 steps: (1) we first remind that lim W 1 (n) = 1 almost surely as n → +∞, thanks to Proposition 2.8, (2) we prove that lim inf n→+∞ W 2 (n) ∧ W 3 (n) > 0 (where x ∧ y denotes the minimum of x and y), (3) we prove that for any w in U := {w ∈ E : w 1 = 1, w 2 w 3 = 0}, the solution of ẏ = F (y) started at w, converges to (1, 1/3, 1/3, 0), (4) we finally apply Corollary 2.6, together with (1), ( 2) and (3) to conclude.
In fact we prove a stronger statement: almost surely for all n large enough, (4.5) To see this, set Note that, conditionally on the event that edge 2 or 3 is reinforced but not both, the one with largest weight is more likely to be reinforced than the other.This implies that the process (Z k ) k 0 taken at its jump times (i.e., the times k = 0 and all k 1 such that Z k = Z k−1 ) stochastically dominates a simple random walk (rSRW n ) n 0 on Z + = {0, 1, . ..} reflected at 0. We let L(n) denote the number of times (Z k ) k 0 returns to zero before time n, i.e., and N (n) the number of jump times of (Z k ) k 0 during its first L(n) excursions out of zero (equivalently the number of times only one of the two edges 2 or 3 is reinforced before the last time before n when the weights of edges 2 and 3 are equal).Then, for all integers n, L(n) is stochastically dominated by the number of returns to the origin of (rSRW k ) k 0 before time N (n).Moreover, by definition, During each excursion out of the origin, the probability to hit level N 5/8 is equal to N −5/8 , by a standard Gambler's ruin estimate.Moreover, by Hoeffding's inequality, for any N 1, the probability for a simple random walk started at level N 5/8 to hit 0 before time N is bounded by 2N exp(−N 1/4 /2).Therefore the probability that (rSRW k ) k 0 returns more than N 3/4 /2 times to the origin before time N is bounded by (1 − N −5/8 ) N 3/4 /2 + 2N exp(−N 1/4 /2) exp(−N 1/8 /4), for all large N .Using the Borel-Cantelli lemma, we deduce that, almost surely for all large N , (rSRW k ) k 0 returns at most N 3/4 /2 times to the origin before time N .This implies that, almost surely for all large n, On the other hand, during each excursion of the process (Z k ) k 0 out of zero, the process (Φ(k)) k 0 increases by at least half the number of jumps made by (Z k ) k 0 during this excursion.It follows that N (n) 2Φ(n), for all n 0, and together with (4.6) and (4.7), this concludes the proof of (4.5), and thus of point (2).

Two parallel paths: proof of Theorem 1.5
Recall that, in Theorem 1.5, the graph G = (V, E) is the (p + q)-path graph of Figure 2. In this section, we assume that min(p, q) 2. We let a 1 , . . ., a p denote the p edges on one of the paths from N to F (ordered from N to F , i.e., a 1 links to N while a p links to F ), and b 1 , . . ., b q denote the edges on the other path from N to F (also ordered from N to F ).
Finally, for all e ∈ E = {a 1 , . . ., a p , b 1 , . . ., b q }, we let W e (n) denote the weight of edge e at time n and set The proof of Theorem 1.5 uses the ODE method, as for the cone graph in the previous section.We roughly follow the same steps here, but there are some important differences.We first show in Section 5.1 that almost surely the limiting set L(W ) of the sequence of normalised weights W (n) := W (n)/(n + 1), is contained in (0, 1] p+q .Then, in Section 5.2, we give an explicit expression for the vector field F appearing in the stochastic approximation satisfied by ( W (n)) n 0 .In this section, we also define a sequence of compact sets (K n ) n 0 in [0, 1] p+q , which we prove to be decreasing.Then in Section 5.3, we show that L(W ) ⊆ K n , for all n 0, and finally in Section 5.4, we prove that the intersection of the K n 's is reduced to a single point, which is precisely the limiting point arising in the statement of Theorem 1.5.We prove here the following result.
Proposition 5.1.-Let p and q be integers such that min(p, q) 2. If G = (V, E) is the (p, q)-path graph, then there exists a constant c p,q > 0, such that almost surely for all e ∈ E, This proposition is proved by induction on k for a k ∈ {a 1 , . . ., a p } and by induction on for b ∈ {b 1 , . . ., b q }.We prove the base case separately in the following lemma.
Proof.-For all n 0, we have (5.1) . Note that, by definition of the model, for all n 0, we have either a p ∈ γ(n + 1) or b q ∈ γ(n + 1) but not both.This implies that, for all n 0, By definition of the model again, we have that a p ∈ γ(n + 1) Thus, (5.1) implies that where we have set Z(n) := W a1 (n)/(n + 1), for all n 0. This implies that (W a1 (n)) n 0 stochastically dominates a G-urn process, where, for all z ∈ [0, 1], Note that G(z) = z if and only if z = 0 or z = 1−1/q (which is positive because q 2, by assumption); furthermore, one can check that G is C 1 and G (0) = q > 1, implying that the normalised G-urn process converges almost surely to c := 1 − 1/q > 0, when n → +∞, by Proposition 2.1.This concludes the proof.
Proof of Proposition 5.1.-We prove the result by induction on k ∈ {1, . . ., p}.The case k = 1 follows from Lemma 5.2.Assume the result holds true for some k < p, i.e., there exists c ∈ (0, 1/2) such that almost surely lim inf W a k (n)/n > c.Let us define the events Then, m E m holds almost surely.Let τ n be the time when edge a k is reinforced for the n-th time (so by definition W a k (τ n ) = n + 1).Note that by definition τ n n for all n 1.Thus, for all n m, on E m , using that W a k−1 (τ n ) τ n + 1 and τ n (n + 1)/c, one has It follows that, almost surely on the event E m , if the weight of a k+1 is zero, then, for all n m, the τ n -th ant makes at least a geometric number of crossings of edge a k , with success probability ν := 1 − c/2, before jumping across edge a k−1 .Consequently, on E m , and for n m, the probability to reinforce edge a k+1 at time τ n is at least , where (X n ) n 0 is a sequence of i.i.d.geometric random variables with parameter ν, and Thus, letting u k+1 (n) = W a k+1 (τ n )/(n + 1), we find with (F i ) i 0 the natural filtration of the process, and where we used that Ex Xn = νx/(1 − (1 − ν)x) for all x ∈ (0, 1).It follows that on the event E m , the process (W a k+1 (τ n )) n m stochastically dominates a G-urn process (starting from W a k+1 (τ m )), with G(x) := x/(x + ν).Since G(x) > x for all x ∈ (0, c/2), it follows from Proposition 2.1 that almost surely lim inf u k+1 (n) c/2.We deduce the induction step, using that by hypothesis, lim sup τ n /n 1/c.

5.2.
The stochastic algorithm and a sequence of decreasing compact subspaces.
Recall that we set W (n) := W (n)/(n + 1), for all n 0. Note also that, by definition of the model, for all n 0, with E as defined in (2.5).Moreover, for all n 0, we have with ξ n+1 some martingale difference and with F as defined in (2.4).More specifically the coordinates of F can be computed explicitly here, and are given by, for all k and such that 1 k p and 1 q, for all w ∈ E , (5.2) where we have defined, for any w ∈ [0, 1] E , m ∈ {a, b}, and s an integer such that 1 s p if m = a, and 1 s q if m = b, Note that, for all k such that 1 k p, S a k (w) = 0 if and only if w ai = 0 for some i such that 1 i k, and for all such that 1 q, S b (w) = 0 if and only if w bi = 0 for some i such that 1 i .To prove Theorem 1.5, we use the ODE method and thus start by studying the solutions of the equation ẏ = F (y).To do so, we define a sequence (K n ) n 0 of decreasing compact subsets of E such that (A) for all n 0, L(W ) ⊆ K n , and (B) the intersection of all these compacts is {w * }, where w * a k = α k and w * b = β , with (α, β) as in Theorem 1.5.We prove (A) in Section (5.3), and (B) in Section 5.4.In the rest of this section, we define the sequence (K n ) n 0 and show that it is decreasing, i.e., that K n+1 ⊂ K n for all n 0. For this we need some additional notation.For all u, v ∈ E , we let and , for all k and such that 1 k p and 1 q.Recall further that by Proposition 5.1, there exists a constant c p,q > 0, such that almost surely (5.4) L(W ) ⊆ {w : w e c p,q , for all e ∈ E}.
We also define w * as the limiting vector appearing in the statement of Theorem 1.5.More precisely, we have (5.5)w * a k = α k , and w * b = β , for all k and such that 1 k p, 1 q, with (α, β) the unique solution of the system (1.1) in (0, 1) 2 (existence and uniqueness of the solution for this system of equations will be proved later, at the end Section 5.4).We then define u (0) and v (0) by , for all k and such that 1 k p, 1 q, with u 0 chosen arbitrarily so that The fact that we choose u 0 min(α, β) entails in particular 0 < u (0) e w * e , for all e ∈ E.
J.É.P. -M., 2022, tome 9 Next we define inductively two sequences (u (n) ) n 0 and (v (n) )) n 0 , by Finally, we define the sequence (K n ) n 0 by e w e v (n) e , for all e ∈ E} (∀n 0).We prove now that this sequence is decreasing, that is, K n+1 ⊂ K n , for all n 0.More precisely, we prove that, for all n 0, We reason by induction on n: first note that, for all k and such that 1 k p and 1 q, b .We now proceed to the induction step and assume that, for some n 1 u (n)  e > u (n−1) e and v (n) e < v (n−1) e for all e ∈ E. We then simply observe that, for all e ∈ E, u e , which proves the induction step, and thus concludes the proofs of (5.6) and (5.7).In other words we just have proved that (K n ) n 0 is indeed a sequence of decreasing sets.
-We use an induction argument.Note first that by (5.4) and the definition of K 0 , one has almost surely L(W ) ⊆ K 0 , using also the hypothesis u 0 c p,q .We now prove the induction step, i.e., that almost surely, if L(W ) ⊆ K n , for some n 0, then also L(W ) ⊆ K n+1 .
To do so, we first look at F a k (w) for 1 k p and w such that u e w e v e for all e ∈ E: we have Thus, if u e w e v e for all e ∈ E and J.É.P. -M., 2022, tome 9 then F a k (w) > 0. Also, for all k such that 1 k p, if u e w e v e for all e ∈ E and then F a k (w) < 0. The same argument leads to for all such that1 q, and if u e w e v e for all e ∈ E. These facts imply that, for all n 0, and for any w ∈ K n , the flow of the ODE ẏ = F (y) started at w converges to K n+1 .Therefore if we already know that L(W ) ⊆ K n , then it means that L(W ) ∩ K n+1 is an attractor of the flow restricted to L(W ).Thus by Proposition 2.5, we deduce that L(W ) ⊆ K n+1 .
Altogether this proves that L(W ) is almost surely included in the intersection of all the K n 's.

5.4.
Identification of the intersection of the K n 's.-We show here that the intersection of the K n 's is reduced to the single point w * , which appears as the limiting vector in the statement of Theorem 1.5 (see also (5.5) above).
Since the sequences (u (n) e ) and (v (n) e ) are all monotonic and bounded, they all converge.We let u * = lim n→+∞ u (n) and v * = lim n→+∞ u (n) .Because H is continuous on E , we have u * = H(u * , v * ), and v * = H(v * , u * ).
The equation u * = H(u * , v * ) can be written as, for all k and such that 1 k p, .
Using that S a 1 (u * ) = u * a1 and S b 1 (u * ) = u * b1 , this implies that (5.8) u * a1 = 1 − S b q (v * ) =: α and u * b1 = 1 − S a p (v * ) =: β, and, for all k and such that 2 k p, 2 q, We first show by induction that this implies u * a k = α k and u * b = β for all k and such that 1 k p and 1 q.Indeed, if for some k such that 1 < k p, and all i such that 1 i < k, u * ai = α i , then , and a straightforward calculation yields that u * a k = α k , as claimed.The proof of u * b = β for all such that 1 q is similar.
Since (u * , v * ) also satisfies the symmetric equation v * = H(v * , u * ), we get that (5.9) v * a1 = 1 − S b q (u * ) =: α and v * b1 = 1 − S a p (u * ) =: β, and v * a k = α k , v * b = β for all k and such that 1 k p and 1 q.Using this into (5.8), and using the definition of S p and S q (see (5.3)), we get Similarly, using the fact that u * a k = α k and u * b = β for all k and such that 1 k p and 1 q, together with Equation (5.9), we get and, similarly, (5.11) For all integer p 2 and x ∈ [0, 1], we let With this notation, we have α = f q (β) and β = f p (α) (and similarly for α and β), and thus We now show that f p is a contraction for all p 2, implying that f p • f q is also a contraction, and thus admits a unique fixed point, which implies α = α (and thus β = β).

6.
The lozenge: proof of Proposition 1.6 First recall that, by Proposition 2.7 and the specificities of the lozenge graph, if we let W (n) = W (n)/(n + 2) (∀n 0), then, for all n 0, W (n) ∈ E , where E := w = (w 1 , w 2 , w 3 , w 4 , w 5 ) ∈ E : w 2 + w 5 = 1, w 1 + w 4 1, w 2 w 1 + w 3 , w 5 w 3 + w 4 , with E as defined in (2.5).The first condition (w 2 + w 5 = 1) is satisfied by W (n) because, by definition of the model, each ant reinforces either edge 2 or edge 5 but not both.The second condition (w 1 + w 4 1) is redundant with the fact that w ∈ E (it is the same condition as π w (N ) 1).The third condition (w 2 w 1 + w 3 ) holds because each ant that reinforces edge 2 also reinforces either edge 1 or edge 3. The fourth condition is the symmetric of the third one.
Remark.-As in Section 4, it is convenient to define W (n) as W (n)/(n + 2) rather than as W (n)/(n + 1).This is because with the former definition, we have W 2 (n) + W 5 (n) = 1 for all n 0, which is not true with the latter definition (although it holds asymptotically as n → +∞).
Moreover, for all n 0, with ξ n+1 some martingale difference, and where F i (w) = p i (w) − w i , with p i (w) = P(e i ∈ γ n+1 | W (n) = w) (note that this probability does not depend on n).The first step in the proof of Proposition 1.6 is to compute these probabilities p i (w), for 1 i 5. A straightforward calculation, which we carry out in the appendix, shows that for all w ∈ E , By symmetry, we also have p 4 (w) = p 1 (w 4 , w 5 , w 3 , w 1 , w 2 ) and p 5 (w) = p 2 (w 4 , w 5 , w 3 , w 1 , w 2 ).
We next prove the following result.
As a corollary we get the following: Proof.-We first show that Φ(t) converges to the set Denote by (Φ i (t)) i=1,...,5 the coordinates of the vector Φ(t), and let By definition, taking the derivative along the flow, we get Our aim is to show that h(t) := max(|u(t)|, |v(t)|) is a Lyapunov function, i.e., that it is decreasing, for all t smaller than the (possibly infinite) time when it reaches 0, and that it converges to 0. To see this, first note that if at some time t, one has 0 v(t) < u(t), then by Lemma 6.2(i), , then by Lemma 6.2, we have u (t) < 0 and v (t) > 0, which entails that h is decreasing in a neighbourhood of t since both its right and left derivatives are negative at this time (it is differentiable if |u(t)| = |v(t)|).Symmetrically, the same holds if u(t) < 0 v(t).Finally when u(t) = v(t) = 0, we see that u (t) = 0 while v (t) = 0, so that in a neighbourhood of t, u(t) = v(t), and thus by the previous argument h is again decreasing in a neighbourhood of t.As a consequence, h is decreasing up to the (possibly infinite) time when it reaches 0, and thus converges.Note also that the previous arguments show that h has a negative right-derivative at any non-zero value, which implies that its only possible limit is zero.This indeed implies that Φ(t) converges to the set H := E ∩ {w 2 = w 1 /(w 1 + w 4 ) = 1/2}, as claimed.
Appendix.Calculating F in the lozenge case: proof of (6.1) We use the same notation as in Section 6.To prove (6.1), we use the electrical networks method (see, e.g.[LL10]).We start by calculating p 2 (w): this is the probability that a random walker on the graph with weights w = (w i ) 1 i 5 , starting from N , crosses edge 2 before crossing edge 5.This is equal to the probability that a walker starting from N reaches F 2 before F 5 on the weighted graph of Figure 6.We decompose p 2 (w) according to the first step of the walker: where C XY (w) is the effective conductance between vertices X and Y when the edge weights are given by w.By definition of effective conductances, the effective conductance of a single edge is its weight.Thus, C P2F2 = w 2 .Moreover, the conductance of two edges in parallel is the sum of their effective conductances, the effective conductance of two edges in series is the inverse of the sum of the inverses of their effective conductances.Using these formulas, we get (see Figure 6  where p 32 (w) (resp.p 35 (w)) is the probability to cross edge 3 before reaching F starting from P 2 (resp.P 5 ), when the edge weights are given by w.Decomposing over the first weight of a random walker starting at P 2 , we get p 32 (w) = w 1 w 1 + w 2 + w 3 • p 3 (w) + w 3 w 1 + w 2 + w 3 , .
J.É.P. -M., 2022, tome 9 5.1.Proof that none of the edges has a limiting weight equal to zero