We study the consequences for top-quark physics of having electron and positron beams available at the LHeC and FCC-he, as was the case in HERA. We show that the asymmetry between top production in $pe^+$ collisions and antitop production in $pe^-$ reactions is sensitive to $|V_{td}|$. By means of detailed parton-level Monte Carlo simulations of single $t$ and $\bar{t}$ production and its backgrounds, we parametrize the asymmetry dependence on $|V_{td}|$ and estimate its uncertainties. Our analysis includes realistic phase-space cuts and machine-learning binary classifiers for background rejection. We thus obtain limits on $|V_{td}|$ that are substantially stronger than current ones and also smaller than current projections for the HL-LHC. We have $|V_{td}|< 1.6 \times |V_{td}^{\text{PDG}}|$ at the LHeC, at 68\% C.L. with $L_\text{int} = 2$/ab.