A community dataset to make tracking better, what do you think?

General
,
1

Hello,

The biggest thing holding back ball and player tracking is training data. I’d like to build a shared dataset from real match footage so tracking gets better for everyone.

You’d optionally contribute footage from your matches, and in return tracking improves for everyone, and down the line, better models on your own games, since the training has fit to your footage.

Many of you have already sent me your games, and some of you authorized me to use them for training. Thank you.

The question I want your opinion on: should this be a public dataset? That way anyone can train on it themselves, labeling is much easier to share (ball + players, eventually field markings), it scales to other sports without me being the bottleneck, and it would make Reco a real reference for the sports community.

This also goes beyond tracking: better data and models are exactly what unlock analytics and highlights, and I don’t think the scale needed is that large. Here’s a video of what I mean: https://www.youtube.com/watch?v=aBVGKoNZQUw

Regarding existing datasets, in my experience, they don’t fit Reco’s constraints: a fixed rig, with the AI running on the raw unstitched footage.

I am concerned about footage with children in it, whether on or around the pitch. There’s also a control trade-off: a public dataset is hard to take back, since anyone can copy it. Kept private, I can delete your footage whenever you ask.

So, public or private, and what would make you comfortable either way?

1 Like
New Release: v0.5.3 is out
4

I think the best option is Full-Person Blurring with Outlines:

Run a pre-processing pipeline that applies a full silhouette blur + high-contrast outline to every detected person (players, referees, background spectators, and children).

This completely solves the privacy/GDPR issue for public datasets while keeping all the spatial and motion data the tracking models need!

I realize this brings extra work and processing overhead for those responsible for maintaining privacy compliance, but it’s the safest way to make the dataset public.

I think a YOLOv8 pipeline can do this.

5

I have thought of this, but I am afraid the blurring might reduce the accuracy of the model.

At least face only blurring might be enough? and people outside the field might be competely blurred it is fine.

6

I’ll start recording games and uploading the footage to YouTube. You’re welcome to use it in the training dataset. I’ve purchased two Action 5 cameras and am currently waiting for the 3D mounts to be printed. What type of footage are you looking for, and are there any specific recording requirements I should follow?

7

privacy/GDPR … you are not the hostage of that rule, the game is open for public, and the user data is not sold (footage).
Think like a football match of UEFA champion leage, the players & the crowd they are not blurred, because there are more than 3 person in a tv view, it’s not more individual.

update:
Photos and Video (AI/Streaming): Filming matches with automated cameras (like Veo) or publishing crowd/player photos requires notices and respect for opt-outs.

update #2:

1 Like
8

During a discussion with an AI agent about this matter, I asked about a synchronized online dataset.

Answer:
Instead of building a public dataset by uploading match videos, have you considered a privacy-first approach?

Reco could train the AI locally on each user’s computer, and only upload the learned model updates (federated learning) or anonymized labels (player positions, ball coordinates, field markings, etc.), never the original footage.

This would allow the community to continuously improve the AI without users having to share their match videos. It would also make clubs and parents much more comfortable contributing, especially for youth football, since the original recordings would never leave their devices.

It seems like a good compromise between building a large community-driven dataset and protecting privacy.

9

I do not think that making people train locally is a viable solution. It would be a nightmare to setup correctly, and it will be hard to make sure the data is of good quality.

2 Likes
10

Hello, I have Insta360 ONE X2, I can mount it where I have now the VEO CAM 3, will that help ? I have to stream in 360 to get all the field…? Have a good day.

11

Hi @Alex_Puiu,
You only need 180 degrees.

I believe your Insta360 can record from one lens only?

For ActionStitch-like videos, the next release (v0.6.0) will allow you to use one 180 degrees and do AI panning inside.

Hopefully this answers your question.

12

TL/DR - In the US, most games are considered public. I fully support keeping the machine learning open and allowing use of the videos. I am not concerned about privacy as most games already have multiple cameras on the sideline.

As 71m363nd3r said, at least in the US, most games are considered public. I am seeing multiple sideline cameras at lacrosse tournaments. Including the NLV camera, there is usually one other camera, a lot of times two. I’ve counted up to five. The more cameras on the sideline, the more Falcons you see. And with that, a parent missing the game from having to stare at their phone to make sure it’s tracking correctly. The more serious club director/coach/parent will be using a Veo or Pixellot.

Point is, these games are already being recorded and being put online. Personlly I put video on Youtube with an unpublished link so you need that link to find the game. Also being unpublished, it allows parents and coaches to download the video freely to make their own clips. But the link can be shared by anyone to everyone. I’m not that concerned about it.

Using a Pixellot for the last two years, I do not have much video for full field games. I am willing to share all of my video for training. I did record three games this summer with the GoPro Mission 1 Pro set to super wide. Think it got all of the field. I can share those videos. I anticipate GoPro will release an ultra-wide lens for it sometime soon that will get the full 180 degrees. I was surprised how well the battery and temperature resistance worked. Recording for 50 minutes 8k @30 fps, used maybe a third of the battery. Only overheated once in the low 80s with direct sunlight. The other two games, I think the Pixellot provided shade for the GoPro. One of those sun shade cases with a fan should work to keep it from overheating during a game in the 90-100 degree range. It would be nice to use one camera and the desktop software for tracking. Suppose that’s for a different post.

13

Support for one camera for tracking is coming in v0.6.0 (assuming Actionstitch-like videos).

I’m reading the rest soon, very interesting

14

The Mission 1 Pro using Superview provides a 159 degree view. Different LLMs agreed that with the correct height, distance from the sideline, and angle of the camera, an entire soccer/football/lacrosse field could be captured. From what I remember, I used the NCAA womens lacrosse field dimensions as it was the largest for multisport turf fields. I will have to look up the chats, but I think 15 feet high, 15 feet from the sideline, and a 25 degree tilt, could capture the entire field and not much else on the sides. Plenty of sideline coverage top and bottom, with minimal wasted space recording the horizon and sky.

15

Just stumbled across a folder of bookmarked youtube lacrosse game videos that were full field. I searched for those when I was vibe coding a tracking system. I got as far as needed to feed it videos for training. Problem was I didn’t have much of my own from using a Pixelott for the last two years. Never got to doing it as I became more interested in your community project. What’s the best way to send that list to you?