I would replace the chain after cooking the new one in a crockpot of wax. It looks like a single speed. If it is, how about a half-link chain that is super strong? Also some jockey/tensioner wheels are narrow/wide. See if that is the case and if it is off by one cog.
They're basic jockey wheels, so that's not it. It's a generic tensioner off Amazon, but I doubt it's somehow responsible for the wiggle. I'll rotate the chainring's position on the spider by a quarter turn and see if that changes anything.
On the other hand, ain't nothing wrong if my tensioner wants to boogie a little while I ride! It's not quite ready for the road, still finishing the build and awaiting a few more parts.