Step Up! My phone told me I can't dance
This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle
What I Built
Step Up! is a web app that tells you whether your salsa steps land on the beat. You prop your phone up, it plays the music, and it watches you dance through the camera. When you finish, you get a score out of 100 and one tip for next time.
The first thing it taught me was about myself. In every session I recorded, my steps landed one count late: I was stepping on 2, 3, 4 instead of 1, 2, 3. I'd been dancing like that without noticing, and nobody at home was going to tell me.
That's the problem it solves. In a class, a teacher can tell you you're off the beat. At home, practising alone between classes, you can't hear it in your own feet, so it's easy to practise the same mistake for weeks.
Who would pay for it: Salsa learners who take a weekly class and practise at home. A class costs around £10 to £15, so a few pounds a month for something that checks your homework is easy to justify. The next step up is dance schools: a teacher could set a tempo as homework and see each student's scores, which is worth a monthly studio licence.
What it does:
- Plays the music. A built-in rhythm (cowbell, clave and conga), a salsa band with piano and bass, or one of three real songs, at 140 to 200 BPM.
- Works hands-free. After you tap once to turn the camera on, you never touch the phone again. Raise both hands above your head to open the hands-up GUI. Your wrists show as yellow rings, and holding one over a button for 1.5s presses it.
- Scores four things. Tempo (are you the right speed?), Pause length (do you hold on 4 and 8?), Timing (does your basic start on the 1?) and On the beat (does each step land within 80ms of the beat, or are you dragging or rushing?).
- Shows your typical basic. The average of every basic you danced, drawn as six dots, green, amber or red. Tap a score to see it on those dots.
- Explains every score. The full analysis goes through the same four scores, and every box ends with one thing to do. It compares your 123 with your 567, shows the pause on the 4 and the 8 separately, and draws every basic you danced.
- Keeps your history. Every session is saved in your browser on your own device, with a "Your score over time" chart. Tap a dot to open that session.
!Image description
My Results after a session: right speed and pause, but still one beat late.
Demo
Try it: luke-ms-devlin.github.io
It runs in your browser with nothing to install. To try it:
1. Prop your phone or laptop 2.5–3m away, low down and tilted up, with your whole body in frame and the light on.
2. Tap the camera button, then raise both hands above your head to open the hands-up GUI.
3. Choose Full sync first (stand still, then march to 16 clicks), then Start.
4. Dance the basic for 30s and read your score.
Code
{% embed Luke-MS-Devlin/salsa-sync %}
The whole app is one index.html file: no build step, no server, no accounts. It's MIT licensed.
How I Built It
The open-source AI at the core is Google's MediaPipe Pose Landmarker, running the pose_landmarker_lite model in the browser with MediaPipe Tasks Vision. It finds 33 points on your body in every camera frame, on your own phone's GPU. I use the hips, knees and ankles. Everything else is plain JavaScript and the Web Audio API.
Getting from "33 dots" to "your step on 5 was 120ms late" took four problems.
1. Two clocks. The music runs on the audio clock, the camera on the video clock, and they don't agree. If they're 50ms apart, every step looks 50ms off, and that's the size of the mistakes I'm trying to measure. Each camera frame gets its timestamp from requestVideoFrameCallback, and each beat is moved onto the same clock with AudioContext.getOutputTimestamp(). To check it, I marched on the spot to a click: the detector found 78 of 80 steps, with a median of 6ms after the beat.
2. What counts as a step. Salsa steps are small. The steps on 2 and 6 are barely a weight change. For each foot I measure how high the ankle is above where it was resting, divided by leg length so it doesn't matter how far you stand from the camera. A 3s "Motion sync" at the start (just stand still) measures how much the points wobble when you're not moving, and sets how high a foot has to lift to count. The step is timed when the foot comes back down to half its peak height, because a step counts when your weight lands, not when your foot is highest.
3. Bluetooth. Bluetooth headphones and speakers play sound around 150–200ms late, and the browser doesn't know. That would make every step look late, even a perfect one. So there's an Audio sync: you march to 16 clicks, the app takes the median gap, and allows for it from then on. On a new device, one Full sync button on the hands-up GUI runs Motion sync and then Audio sync, and it's highlighted yellow with "do this first" until you've done it.
4. Scores that agree with each other. My first results screen showed percentages and charts, and the numbers sometimes contradicted each other. Now everything comes from your typical basic, the average position of each of the six steps. The words, points and colour of each score come from that one basic, so a green tile never says "Too short". The tip is the first score that fails, so a dancer who's a count late is told "Start on the 1" before anything else. The summary is the quick answer; the full analysis underneath shows where it came from, like a short pause on the 4 hiding behind a long one on the 8.
Things I learned the hard way:
- A menu you can't touch has to stay calm. You're 3m from your phone, so the hands-up GUI used to put a big red "Step back. I can't see your feet." on screen and say it out loud. Now "Can't see you" and "Can't see your feet" are a small silent notice, and the buttons stay a button-height up from the bottom of the screen.
- The model needs your face. I tried filming just my legs from a low camera, and it stopped finding me. MediaPipe's detector uses your face to find where your body is. The fix is a low camera tilted up, head to feet in frame.
- Light changes everything. One late-night session scored nothing. The webcam had halved its frame rate in the dim room (16fps down to about 8fps), so short steps fell between frames. With the main light on it was fine.
For music, the built-in rhythm and band are generated note by note with Web Audio. The three songs are Creative Commons tracks from the Free Music Archive. A live band drifts slightly in tempo, so I found the time of every beat in each song with a beat tracker, and the app scores you against those real beat times.
Why Does Open Innovation Matter?
Privacy. This app films people dancing in their living rooms. With MediaPipe running in the browser, the video never leaves the phone. There's nothing to upload, so there's nothing to leak. I don't think many people would dance in front of a camera that streams to a server.
Cost. A closed vision API charges per frame. At 15–30 frames a second, a 30s session is up to 900 calls, and a keen learner does several a day. Local inference costs nothing per session, which is the only reason a cheap or free version is possible at all.
Speed and offline use. There's no network round trip, so each frame is processed as it arrives, and once the page and model have loaded, a session doesn't need the internet.
I could see how it works. MediaPipe's model and documentation are open, which is how I found out why it lost me when I filmed only my legs: the detector is built on a face detector. With a black-box API I'd have been guessing.
Open all the way down. The music is Creative Commons, and the app is MIT. A dance teacher who wants a different tempo, a different style, or their own counts can fork it.
My Agent Session
I built Step Up! over three days with Claude Code as my coding assistant. I decided what the app should do and how it should look, and tested every version by dancing in front of my phone and laptop. Claude wrote the code and analysed the session data I exported, for example spotting that I dance one count late and that low light was halving my camera's frame rate. Every change came as a pull request with a preview link, and I tried it out before merging it myself: 46 pull requests in total.
Credits
- Body tracking: MediaPipe by Google (Apache 2.0).
- Songs from the Free Music Archive: "Cupido" and "Vamos A Tocar Sonero" by Frank Guerrero y Su Grupo Aché (CC BY-NC-ND 3.0, unchanged), and "La Mujer de Antonio" by SONGO 21 (CC BY-NC-SA 3.0, tempo-changed). Both are non-commercial licences, so a paid version would need licensed music.
- Beat tracking for the songs based on Ellis (2007), "Beat Tracking by Dynamic Programming".
Detecta deepfakes políticos en 2026 con herramientas gratuitas
Deepfakes de líderes mundiales en 2026: cómo detectarlos en tiempo real con herramientas open‑source
---
Introducción
En los últimos meses Google Trends ha mostrado picos “Breakout” en búsquedas como “deepfake presidente” o “video falso elecciones”. Cada vez son más los clips falsos que aparecen en redes sociales y que ponen en riesgo la credibilidad de la información política. En este artículo verás, en menos de 5 min, cómo funciona la tecnología que genera esos vídeos, cuáles son los casos más virales de 2026 y, lo más importante, cómo montar un detector en tiempo real usando solo software libre.
---
1. Tecnologías que impulsan los deepfakes de 2026
| Tipo de modelo | Ventajas | Desventajas | Herramientas open‑source más usadas (2026) |
|----------------|----------|-------------|--------------------------------------------|
| GAN (Generative Adversarial Networks) | Rápidos de entrenar, buen nivel de detalle estático | Artefactos visibles en bordes, poca coherencia temporal | DeepFaceLab, Faceswap |
| Difusión (Stable Diffusion Video, DDIM) | Vídeos suaves, menos ruido, sincronía labial avanzada | Necesitan GPU potente y más tiempo de inferencia | Stable‑Video‑Diffusion, AnimateDiff |
| Hybrid (GAN + Difusión + Audio‑Lip‑Sync) | Calidad casi fotográfica, audio perfectamente alineado | Complejidad de pipeline, consumo energético alto | Sora‑Pipeline, Audio2Face |
En 2026 la combinación difusión + audio‑lip‑sync es la que está detrás de la mayoría de los deepfakes políticos virales (por ejemplo, el falso discurso de la presidenta de Brasil en el G20).
---
2. Casos virales de 2026
| Fecha | Líder | Tema del vídeo | Plataforma donde se viralizó | Comentario de detección |
|-------|-------|----------------|------------------------------|--------------------------|
| 12 mar 2026 | Ursula von der Leyen (UE) | “Anuncio de nuevo impuesto a la IA” | TikTok (8 M visualizaciones) | Detectado por DeepDetect por parpadeo irregular |
| 27 abr 2026 | Joe Biden (EE. UU) | “Mensaje de apoyo a la reforma migratoria” | Twitter (12 M retweets) | Modelo de difusión con audio‑lip‑sync, sin artefactos visibles |
| 03 may 2026 | Xi Jinping (China) | “Declaración sobre la crisis climática” | Weibo (15 M vistas) | Detectado por análisis de compresión JPEG (artefactos en fondo) |
| 19 jun 2026 | Narendra Modi (India) | “Invitación a elecciones anticipadas” | WhatsApp (cadenas de 2 M) | Señalado por DeepTrace por inconsistencias en la luz del fondo |
---
3. Monta tu propio detector en tiempo real (guía paso a paso)
A continuación tienes un pipeline mínimo que puedes ejecutar en una máquina con GPU (NVIDIA RTX 3080 o superior). Usa únicamente librerías gratuitas y está listo para procesar vídeos en directo desde una URL o archivo local.
3.1. Requisitos
# Sistema operativo: Ubuntu 22.04 LTS (también funciona en WSL2)
# Python >= 3.10
# CUDA 12.x + cuDNN
Instala los paquetes necesarios:
# 1. Crear entorno virtual
python3 -m venv venv_deepdetect
source venv_deepdetect/bin/activate
# 2. Instalar PyTorch con soporte CUDA
pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu121
# 3. Librerías de detección
pip install opencv-python ffmpeg-python tqdm \
transformers==4.41.0 \
diffusers==0.27.0 \
facenet-pytorch==2.5.2 \
numpy==1.26.4
3.2. Descargar los modelos pre‑entrenados
# Modelo de detección de artefactos de compresión (DeepDetect)
git clone https://github.com/DeepDetectAI/deepdetect.git
cd deepdetect
git checkout v2.1 # versión optimizada para vídeo
python -m pip install -e .
# Modelo de análisis de parpadeo (BlinkCheck) basado en MediaPipe
git clone https://github.com/google/mediapipe.git
cd mediapipe
python -m pip install .
3.3. Código de detección (ejemplo completo)
import cv2, ffmpeg, numpy as np, torch
from deepdetect import DeepDetectModel
from mediapipe import solutions as mp_solutions
from tqdm import tqdm
# Cargar modelos
dd_model = DeepDetectModel('weights/compression_artifacts.pt')
blink = mp_solutions.face_mesh.FaceMesh(
static_image_mode=False,
max_num_faces=1,
refine_landmarks=True,
min_detection_confidence=0.5,
min_tracking_confidence=0.5,
)
def analizar_video(ruta):
"""Devuelve un puntaje de autenticidad entre 0 (fake) y 1 (real)."""
# 1️⃣ Extraer frames (1 fps para rapidez)
probe = ffmpeg.probe(ruta)
fps = eval(probe['streams'][0]['r_frame_rate'])
total_frames = int(probe['streams'][0]['duration'] * fps)
step = int(fps) # 1 frame por segundo
scores = []
for i in tqdm(range(0, total_frames, step)):
out, _ = (
ffmpeg.input(ruta, ss=i/fps)
.filter('scale', 640, -1)
.output('pipe:', vframes=1, format='image2', vcodec='png')
.run(capture_stdout=True, capture_stderr=True)
)
img = cv2.imdecode(np.frombuffer(out, np.uint8), cv2.IMREAD_COLOR)
# 2️⃣ Artefactos de compresión
comp_score = dd_model.predict(img)
# 3️⃣ Parpadeo anómalo
results = blink.process(cv2.cvtColor(img, cv2.COLOR_BGR_RGB))
blink_score = 0.0
if results.multi_face_landmarks:
# cálculo muy simplificado: distancia entre párpados
landmarks = results.multi_face_landmarks[0].landmark
eye_dist = np.linalg.norm(
np.array([landmarks[159].x, landmarks[159].y]) -
np.array([landmarks[145].x, landmarks[145].y])
)
blink_score = 1.0 - np.clip(eye_dist * 30, 0, 1) # 1 = normal
# 4️⃣ Combinar
final = 0.6 * comp_score + 0.4 * blink_score
scores.append(final)
return float(np.mean(scores))
if __name__ == '__main__':
url = 'https://example.com/video.mp4' # o ruta local
puntaje = analizar_video(url)
print(f'🔎 Puntaje de autenticidad: {puntaje:.2f}')
if puntaje < 0.5:
print('⚠️ Probable deepfake')
else:
print('✅ Parece genuino')
> Cómo funciona
> 1. Compresión – El modelo DeepDetect reconoce patrones de cuantización típicos de los algoritmos de exportación de vídeo sintético.
> 2. Parpadeo – Los deepfakes basados en GAN suelen generar parpadeos irregulares; MediaPipe mide la apertura de los párpados y detecta anomalías.
> 3. Ponderación – Se combinan ambos indicadores (60 % compresión, 40 % parpadeo) para obtener un puntaje global.
3.4. Integración rápida en una página web
Si prefieres un widget HTML/JS que cualquiera pueda usar sin instalar nada, copia este bloque en tu sitio y enlaza el script deepdetect-widget.js (disponible en el repositorio del artículo).
<div id="deepfake-checker" style="max-width:480px;margin:auto;"></div>
<script src="https://cdn.jsdelivr.net/gh/tuusuario/deepdetect-widget@v1.0/deepdetect-widget.js"></script>
<script>
DeepFakeWidget.init({
container: '#deepfake-checker',
apiEndpoint: 'https://api.tuservidor.com/analyze', // tu backend con el script Python anterior
language: 'es'
});
</script>
El widget permite arrastrar un archivo o pegar una URL y muestra el puntaje en segundos.
---
4. Checklist rápido de verificación (para periodistas y ciudadanos)
---
Herramienta mencionada: Heroku
On-Device Computer Vision in React Native: Auto-Aligning Progress Photos with MediaPipe and Expo
---
title: "On-Device Computer Vision in React Native: Auto-Aligning Progress Photos with MediaPipe and Expo"
published: true
description: "Derive a two-eye alignment transform, integrate on-device landmarks with Expo, and render progress timelapses without uploading photos."
tags: reactnative, expo, computervision, mobile
canonical_url: phasesapp.vercel.app
---
A person can take thirty photos of the same face and still produce a timelapse that looks like camera shake. On Monday they stand closer to the mirror. On Tuesday they tilt their phone. By Friday they frame their eyes near the top of the image.
For fitness, skincare, and haircut timelines, viewers need to see the change in the person without chasing their face around the screen. In Phases, the useful visual reference is the eye line: keep both eyes at the same output coordinates across the sequence.
This article develops the React Native and Expo architecture for that workflow: detect landmarks on the device, compute a two-point transform, retain originals in local storage, and render a reel without a rendering server.
Implementation scope: the public Phases site currently describes a browser app that users can add to their home screen. It also describes consent-based analytics. The native Expo architecture and zero-telemetry policy below describe the reference design for this article, rather than a verified description of that deployed browser build. The project link is Ishannaik/phases; the implementation examples below stand on their own.
Separate the photograph from its alignment
A capture guide helps the user repeat their pose, but it cannot guarantee matching camera distance or roll. I would keep the original photo and store its alignment metadata beside it. That lets the user change the reel's framing without reshooting or accumulating JPEG compression from repeated edits.
flowchart TD
A[Camera capture] --> B[Normalize orientation and mirroring]
B --> C[On-device facial landmark detection]
C --> D{Usable eye anchors?}
D -->|Yes| E[Compute affine matrix]
D -->|No| F[Ask user to retake or mark anchors]
B --> G[Original photo in app filesystem]
E --> H[SQLite metadata and transform]
G --> I[Local Canvas or WebGL composite]
H --> I
I --> J[On-device video encoder]
J --> K[Timelapse reel on device]
G -. Optional user-enabled backup .-> L[Cloud backup]
The engineer should treat capture, detection, and rendering as separate steps. After a successful detection, the renderer needs the photo URI and six transform coefficients. It does not need another inference pass each time the user plays the timeline.
For a native app identified as io.phases.app, use its app-private filesystem for photos and SQLite for timestamps, dimensions, eye coordinates, and alignment settings. io.phases.app is an application identifier, not a portable filesystem path. Android and iOS choose the sandbox's physical location.
With Expo, expo-file-system provides document storage, and expo-sqlite provides a persistent database. Store large image files in the filesystem rather than putting base64 photo strings into database rows.
Choose eye anchors your detector can supply
With MediaPipe, you can derive eye anchors from a face mesh. The older Face Mesh API offers iris refinement through refine_landmarks; the current Face Landmarker task provides a denser landmark result. Google documents 478 landmarks for its current model bundle.
!MediaPipe canonical face mesh with landmark indices and eye-region topology
Google's canonical face model UV visualization. This repository asset illustrates landmark topology, not a Phases screenshot or a detector result from this article.
ML Kit can provide eye landmarks through its face detection API, but an eye landmark is not equivalent to a measured pupil center. If you choose ML Kit, name those anchors eye centers. If you need iris-center estimates, choose a detector configuration that supplies iris landmarks and document your index mapping against that model version.
Even an iris center gives an estimate of the visible iris, not a clinical measurement of pupil position. For framing a selfie sequence, the distinction matters because eye movement can introduce jitter. A user looking left and then right can change the anchors without moving their head. A consistent gaze and capture guide help. Eye-contour centers offer another anchor convention when gaze changes dominate the result.
I would run detection on each captured still image, with one intended face. A low-resolution preview can guide capture, but I would calculate the saved transform from the final image. Preview crops and final camera output often use different coordinate systems.
Before measuring the eye vector:
1. Decode the photograph into a consistent upright orientation.
2. Apply the chosen mirror policy to both pixels and landmarks.
3. Convert detector coordinates into source-image pixels.
4. Preserve consistent eye identities across photographs.
For a detector that returns normalized coordinates, multiply x by image width and y by image height. Measuring an angle in normalized coordinates distorts it whenever width and height differ.
Two anchors determine rotation, scale, and translation
Let the source eye anchors be:
$$
p_1=(u_1,v_1),\qquad p_2=(u_2,v_2).
$$
Choose fixed normalized output anchors:
$$
q_1=(x_1,y_1),\qquad q_2=(x_2,y_2).
$$
For horizontal eyes, choose $y_1=y_2$. In an output with width $W$ and height $H$, convert those targets to pixels:
$$
Q_1=(Wx_1,Hy_1),\qquad Q_2=(Wx_2,Hy_2).
$$
For example, on a 1080 by 1920 portrait canvas, (0.38, 0.35) and (0.62, 0.35) place the eyes at (410.4, 672) and (669.6, 672). Their separation is 259.2 pixels. These are illustrative framing choices, not measured Phases defaults.
SOURCE PHOTO OUTPUT CANVAS
p2 * Q1 *=============* Q2
/ fixed eye axis
/ d
p1 *
tilted eye axis horizontal eye axis
variable separation fixed separation
translate + rotate + uniform scale
Define the source and target eye vectors:
$$
d=p_2-p_1,\qquad D=Q_2-Q_1.
$$
Then calculate:
$$
s=\frac{\lVert D\rVert}{\lVert d\rVert},\qquad
\theta=\operatorname{atan2}(D_y,D_x)-\operatorname{atan2}(d_y,d_x).
$$
Use the source midpoint $m=(p_1+p_2)/2$ and target midpoint $M=(Q_1+Q_2)/2$. For any source pixel $p$:
$$
T(p)=sR(\theta)(p-m)+M,
$$
where:
$$
R(\theta)=
\begin{bmatrix}
\cos\theta & -\sin\theta\
\sin\theta & \cos\theta
\end{bmatrix}.
$$
You can write this as a homogeneous affine matrix:
$$
\begin{bmatrix}
x'\y'\1
\end{bmatrix}
=
\begin{bmatrix}
a&c&e\b&d&f\0&0&1
\end{bmatrix}
\begin{bmatrix}
x\y\1
\end{bmatrix}.
$$
Two point pairs determine this similarity transform, which uses uniform scale, rotation, and translation. They do not determine an arbitrary six-parameter affine transform with shear and independent axis scales. A similarity transform belongs to the affine family and preserves the face's proportions.
For a horizontal target eye line, $\theta$ cancels the source eye angle. The coordinate system uses positive y downward, as Canvas does; calculate both angles in that same system.
Calculate the Canvas coefficients in TypeScript
The following reference implementation accepts source anchors in pixels and target anchors in normalized output coordinates. It returns the six values in the order that Canvas expects for setTransform(a, b, c, d, e, f).
type Point = Readonly<{ x: number; y: number }>;
type Affine = [number, number, number, number, number, number];
function eyeAlignment(
source1: Point,
source2: Point,
target1: Point,
target2: Point,
width: number,
height: number,
): Affine {
const values = [
source1.x, source1.y, source2.x, source2.y,
target1.x, target1.y, target2.x, target2.y,
width, height,
];
if (!values.every(Number.isFinite) || width <= 0 || height <= 0) {
throw new Error("Alignment needs finite coordinates and positive dimensions");
}
if ([target1.x, target1.y, target2.x, target2.y]
.some(value => value < 0 || value > 1)) {
throw new Error("Target anchors must lie within normalized output bounds");
}
const sx = source2.x - source1.x;
const sy = source2.y - source1.y;
const tx = (target2.x - target1.x) * width;
const ty = (target2.y - target1.y) * height;
const sourceDistance = Math.hypot(sx, sy);
const targetDistance = Math.hypot(tx, ty);
if (sourceDistance < 1 || targetDistance < 1) {
throw new Error("Eye anchors must be distinct");
}
const scale = targetDistance / sourceDistance;
const angle = Math.atan2(ty, tx) - Math.atan2(sy, sx);
const a = scale * Math.cos(angle);
const b = scale * Math.sin(angle);
const c = -b;
const d = a;
const mx = (source1.x + source2.x) / 2;
const my = (source1.y + source2.y) / 2;
const qx = (target1.x + target2.x) * width / 2;
const qy = (target1.y + target2.y) * height / 2;
const e = qx - a * mx - c * my;
const f = qy - b * mx - d * my;
return [a, b, c, d, e, f];
}
function drawAlignedFrame(
ctx: CanvasRenderingContext2D,
image: CanvasImageSource,
imageWidth: number,
imageHeight: number,
matrix: Affine,
): void {
ctx.save();
try {
ctx.setTransform(1, 0, 0, 1, 0, 0);
ctx.clearRect(0, 0, ctx.canvas.width, ctx.canvas.height);
ctx.fillStyle = "#000";
ctx.fillRect(0, 0, ctx.canvas.width, ctx.canvas.height);
ctx.setTransform(...matrix);
ctx.drawImage(image, 0, 0, imageWidth, imageHeight);
} finally {
ctx.restore();
}
}
Use imageWidth and imageHeight from the same oriented image coordinate system as the source anchors. The code assumes the caller owns the canvas, with no inherited clipping region.
The one-pixel check prevents division by a degenerate eye vector. Production acceptance needs a stronger, image-relative criterion: a two-pixel eye separation on a large image can still produce an unusable scale. Check confidence, face size, and acceptable scale before saving alignment metadata.
To check the matrix mathematically, apply it to each source anchor. You should recover the corresponding target pixel coordinate:
a*u1 + c*v1 + e = W*x1
b*u1 + d*v1 + f = H*y1
a*u2 + c*v2 + e = W*x2
b*u2 + d*v2 + f = H*y2
These equations provide useful unit-test assertions with a floating-point tolerance. They describe invariants, not test results from a running Phases build.
React Native needs a rendering boundary
React Native does not expose a browser DOM Canvas as a built-in drawing surface. The CanvasRenderingContext2D example belongs in a browser or a WebView. In a native renderer, adapt the same coefficients to that renderer's matrix convention.
You can use a WebView-hosted Canvas for a shared web rendering path, or use a native GPU drawing surface. With WebGL, upload a photo texture and apply the transform while drawing a quad. Account for the renderer's origin, texture orientation, and matrix layout; copying Canvas coefficients into an unrelated matrix API can invert or transpose the result.
For MediaPipe or ML Kit integrations that require native code outside Expo Go, create an Expo development build. Installing a JavaScript wrapper cannot add its native implementation to the Expo Go binary. Expo documents this boundary in its development build guidance.
Keep decoded image data near the detector and renderer. Passing a full-resolution base64 image through repeated JavaScript bridge messages adds allocation and copying. Transfer a URI or native image handle where the integration supports it.
Render frames locally, then encode locally
Drawing aligned photos on a canvas gives you frames. To export a video file, you also need an encoder and a container writer.
In a native implementation, connect the rendering surface to an on-device encoder. In a browser implementation, select a supported local recording or encoding API and handle device-specific codec availability. Do not assume a Canvas screenshot sequence equals an MP4 export.
For an export with one photograph per time interval, calculate presentation timestamps from the intended playback cadence. Processing speed should not determine the video's timing. A slow decode must delay export work, not stretch one day into a longer frame.
Memory makes sequential processing important. One 1080 by 1920 RGBA frame takes 8,294,400 bytes, about 7.9 MiB before renderer and decoder overhead. Holding 365 such frames would consume about 2.8 GiB in pixel buffers alone. Decode a bounded number of photos, render a frame, submit it to the encoder, and release resources as the encoder permits.
You should also choose how to handle uncovered canvas areas. Rotation can expose triangles at the corners; scaling cannot recover pixels outside the original capture. Use a consistent background, crop, or capture margin. Avoid changing the crop per frame, since that reintroduces apparent zoom.
Local storage and an optional backup path
A useful local record links a photo URI to its capture time, oriented dimensions, source eye anchors, detector version, and alignment settings version. Retain the originals so you can recalculate transforms after changing the target eye positions.
Coordinate filesystem writes with database commits. Write the image into durable document storage before committing a row that references it. If the database commit fails, remove the new file or reconcile it on the next launch. SQLite cannot roll back a separate filesystem write.
The reference design uses a zero-telemetry policy: no analytics events, session replay, or automatic diagnostic uploads. Developers must apply that policy to crash-reporting SDKs and dependencies as well as application code. Local inference does not by itself guarantee zero network traffic.
For offline detection on first use, package the model with the app. A model download on first launch introduces a network prerequisite even if subsequent inference stays local.
Keep cloud backup outside the alignment and rendering path. A user who declines backup should still capture, align, preview, and export. If they enable backup, explain which originals and metadata you upload, request the provider's permissions, and make local deletion versus remote deletion explicit. App-private storage also does not imply encryption or exclusion from operating-system backups; configure those policies for the threat model you support.
The public Phases FAQ describes optional Google Drive backup and analytics that remain off until consent. Readers evaluating the deployed app should use that published policy rather than the stricter native reference policy above.
Eye alignment has a defined limit
A two-eye transform corrects translation, camera roll, and apparent face scale. It cannot correct a person turning their head, pitching their chin, changing expression, or photographing themselves under a different light source. On a fitness timeline, the eyes also cannot guarantee matching shoulder position or body pose.
I would keep those constraints visible in capture UX: guide the user toward the same gaze, pose, and lighting, then use alignment to absorb small framing errors. For a failed detection or an occluded eye, offer a retake or manual anchors. Do not invent a plausible transform and label it a successful alignment.
Phases' core visual goal is concrete: put the eye axis in the same place across days. With consistent image coordinates, a four-degree-of-freedom transform, and a local rendering pipeline, you can achieve that framing without sending a user's progress photos to an inference or rendering service.
References
- Phases and its published FAQ
- Phases project repository
- MediaPipe Face Landmarker model and task documentation
- MediaPipe legacy Face Mesh and iris refinement
- ML Kit face detection concepts
- Expo SQLite
- Expo filesystem
Written by Ishan Naik.
2 reactions · 2 comments
Mentions
Step Up! My phone told me I can't dance This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle What I Built Step Up! is a web app that tells you whether your salsa steps land on the beat. You prop your phone up, it plays the music, and it watches you dance through the camera. When you finish, you get a score out of 100 and one tip for next time. The first thing it taught me was about myself. In every session I recorded, my steps landed one count late: I was stepping on 2, 3, 4 instead of 1, 2, 3. I'd been dancing like that without noticing, and nobody at home was going to tell me. That's the problem it solves. In a class, a teacher can tell you you're off the beat. At home, practising alone between classes, you can't hear it in your own feet, so it's easy to practise the same mistake for weeks. Who would pay for it: Salsa learners who take a weekly class and practise at home. A class costs around £10 to £15, so a few pounds a month for something that checks your homework is easy to justify. The next step up is dance schools: a teacher could set a tempo as homework and see each student's scores, which is worth a monthly studio licence. What it does: - Plays the music. A built-in rhythm (cowbell, clave and conga), a salsa band with piano and bass, or one of three real songs, at 140 to 200 BPM. - Works hands-free. After you tap once to turn the camera on, you never touch the phone again. Raise both hands above your head to open the hands-up GUI. Your wrists show as yellow rings, and holding one over a button for 1.5s presses it. - Scores four things. Tempo (are you the right speed?), Pause length (do you hold on 4 and 8?), Timing (does your basic start on the 1?) and On the beat (does each step land within 80ms of the beat, or are you dragging or rushing?). - Shows your typical basic. The average of every basic you danced, drawn as six dots, green, amber or red. Tap a score to see it on those dots. - Explains every score. The full analysis goes through the same four scores, and every box ends with one thing to do. It compares your 123 with your 567, shows the pause on the 4 and the 8 separately, and draws every basic you danced. - Keeps your history. Every session is saved in your browser on your own device, with a "Your score over time" chart. Tap a dot to open that session. !Image description My Results after a session: right speed and pause, but still one beat late. Demo Try it: luke-ms-devlin.github.io It runs in your browser with nothing to install. To try it: 1. Prop your phone or laptop 2.5–3m away, low down and tilted up, with your whole body in frame and the light on. 2. Tap the camera button, then raise both hands above your head to open the hands-up GUI. 3. Choose Full sync first (stand still, then march to 16 clicks), then Start. 4. Dance the basic for 30s and read your score. Code {% embed Luke-MS-Devlin/salsa-sync %} The whole app is one index.html file: no build step, no server, no accounts. It's MIT licensed. How I Built It The open-source AI at the core is Google's MediaPipe Pose Landmarker, running the pose_landmarker_lite model in the browser with MediaPipe Tasks Vision. It finds 33 points on your body in every camera frame, on your own phone's GPU. I use the hips, knees and ankles. Everything else is plain JavaScript and the Web Audio API. Getting from "33 dots" to "your step on 5 was 120ms late" took four problems. 1. Two clocks. The music runs on the audio clock, the camera on the video clock, and they don't agree. If they're 50ms apart, every step looks 50ms off, and that's the size of the mistakes I'm trying to measure. Each camera frame gets its timestamp from requestVideoFrameCallback, and each beat is moved onto the same clock with AudioContext.getOutputTimestamp(). To check it, I marched on the spot to a click: the detector found 78 of 80 steps, with a median of 6ms after the beat. 2. What counts as a step. Salsa steps are small. The steps on 2 and 6 are barely a weight change. For each foot I measure how high the ankle is above where it was resting, divided by leg length so it doesn't matter how far you stand from the camera. A 3s "Motion sync" at the start (just stand still) measures how much the points wobble when you're not moving, and sets how high a foot has to lift to count. The step is timed when the foot comes back down to half its peak height, because a step counts when your weight lands, not when your foot is highest. 3. Bluetooth. Bluetooth headphones and speakers play sound around 150–200ms late, and the browser doesn't know. That would make every step look late, even a perfect one. So there's an Audio sync: you march to 16 clicks, the app takes the median gap, and allows for it from then on. On a new device, one Full sync button on the hands-up GUI runs Motion sync and then Audio sync, and it's highlighted yellow with "do this first" until you've done it. 4. Scores that agree with each other. My first results screen showed percentages and charts, and the numbers sometimes contradicted each other. Now everything comes from your typical basic, the average position of each of the six steps. The words, points and colour of each score come from that one basic, so a green tile never says "Too short". The tip is the first score that fails, so a dancer who's a count late is told "Start on the 1" before anything else. The summary is the quick answer; the full analysis underneath shows where it came from, like a short pause on the 4 hiding behind a long one on the 8. Things I learned the hard way: - A menu you can't touch has to stay calm. You're 3m from your phone, so the hands-up GUI used to put a big red "Step back. I can't see your feet." on screen and say it out loud. Now "Can't see you" and "Can't see your feet" are a small silent notice, and the buttons stay a button-height up from the bottom of the screen. - The model needs your face. I tried filming just my legs from a low camera, and it stopped finding me. MediaPipe's detector uses your face to find where your body is. The fix is a low camera tilted up, head to feet in frame. - Light changes everything. One late-night session scored nothing. The webcam had halved its frame rate in the dim room (16fps down to about 8fps), so short steps fell between frames. With the main light on it was fine. For music, the built-in rhythm and band are generated note by note with Web Audio. The three songs are Creative Commons tracks from the Free Music Archive. A live band drifts slightly in tempo, so I found the time of every beat in each song with a beat tracker, and the app scores you against those real beat times. Why Does Open Innovation Matter? Privacy. This app films people dancing in their living rooms. With MediaPipe running in the browser, the video never leaves the phone. There's nothing to upload, so there's nothing to leak. I don't think many people would dance in front of a camera that streams to a server. Cost. A closed vision API charges per frame. At 15–30 frames a second, a 30s session is up to 900 calls, and a keen learner does several a day. Local inference costs nothing per session, which is the only reason a cheap or free version is possible at all. Speed and offline use. There's no network round trip, so each frame is processed as it arrives, and once the page and model have loaded, a session doesn't need the internet. I could see how it works. MediaPipe's model and documentation are open, which is how I found out why it lost me when I filmed only my legs: the detector is built on a face detector. With a black-box API I'd have been guessing. Open all the way down. The music is Creative Commons, and the app is MIT. A dance teacher who wants a different tempo, a different style, or their own counts can fork it. My Agent Session I built Step Up! over three days with Claude Code as my coding assistant. I decided what the app should do and how it should look, and tested every version by dancing in front of my phone and laptop. Claude wrote the code and analysed the session data I exported, for example spotting that I dance one count late and that low light was halving my camera's frame rate. Every change came as a pull request with a preview link, and I tried it out before merging it myself: 46 pull requests in total. Credits - Body tracking: MediaPipe by Google (Apache 2.0). - Songs from the Free Music Archive: "Cupido" and "Vamos A Tocar Sonero" by Frank Guerrero y Su Grupo Aché (CC BY-NC-ND 3.0, unchanged), and "La Mujer de Antonio" by SONGO 21 (CC BY-NC-SA 3.0, tempo-changed). Both are non-commercial licences, so a paid version would need licensed music. - Beat tracking for the songs based on Ellis (2007), "Beat Tracking by Dynamic Programming".
Detecta deepfakes políticos en 2026 con herramientas gratuitas Deepfakes de líderes mundiales en 2026: cómo detectarlos en tiempo real con herramientas open‑source --- Introducción En los últimos meses Google Trends ha mostrado picos “Breakout” en búsquedas como “deepfake presidente” o “video falso elecciones”. Cada vez son más los clips falsos que aparecen en redes sociales y que ponen en riesgo la credibilidad de la información política. En este artículo verás, en menos de 5 min, cómo funciona la tecnología que genera esos vídeos, cuáles son los casos más virales de 2026 y, lo más importante, cómo montar un detector en tiempo real usando solo software libre. --- 1. Tecnologías que impulsan los deepfakes de 2026 | Tipo de modelo | Ventajas | Desventajas | Herramientas open‑source más usadas (2026) | |----------------|----------|-------------|--------------------------------------------| | GAN (Generative Adversarial Networks) | Rápidos de entrenar, buen nivel de detalle estático | Artefactos visibles en bordes, poca coherencia temporal | DeepFaceLab, Faceswap | | Difusión (Stable Diffusion Video, DDIM) | Vídeos suaves, menos ruido, sincronía labial avanzada | Necesitan GPU potente y más tiempo de inferencia | Stable‑Video‑Diffusion, AnimateDiff | | Hybrid (GAN + Difusión + Audio‑Lip‑Sync) | Calidad casi fotográfica, audio perfectamente alineado | Complejidad de pipeline, consumo energético alto | Sora‑Pipeline, Audio2Face | En 2026 la combinación difusión + audio‑lip‑sync es la que está detrás de la mayoría de los deepfakes políticos virales (por ejemplo, el falso discurso de la presidenta de Brasil en el G20). --- 2. Casos virales de 2026 | Fecha | Líder | Tema del vídeo | Plataforma donde se viralizó | Comentario de detección | |-------|-------|----------------|------------------------------|--------------------------| | 12 mar 2026 | Ursula von der Leyen (UE) | “Anuncio de nuevo impuesto a la IA” | TikTok (8 M visualizaciones) | Detectado por DeepDetect por parpadeo irregular | | 27 abr 2026 | Joe Biden (EE. UU) | “Mensaje de apoyo a la reforma migratoria” | Twitter (12 M retweets) | Modelo de difusión con audio‑lip‑sync, sin artefactos visibles | | 03 may 2026 | Xi Jinping (China) | “Declaración sobre la crisis climática” | Weibo (15 M vistas) | Detectado por análisis de compresión JPEG (artefactos en fondo) | | 19 jun 2026 | Narendra Modi (India) | “Invitación a elecciones anticipadas” | WhatsApp (cadenas de 2 M) | Señalado por DeepTrace por inconsistencias en la luz del fondo | --- 3. Monta tu propio detector en tiempo real (guía paso a paso) A continuación tienes un pipeline mínimo que puedes ejecutar en una máquina con GPU (NVIDIA RTX 3080 o superior). Usa únicamente librerías gratuitas y está listo para procesar vídeos en directo desde una URL o archivo local. 3.1. Requisitos # Sistema operativo: Ubuntu 22.04 LTS (también funciona en WSL2) # Python >= 3.10 # CUDA 12.x + cuDNN Instala los paquetes necesarios: # 1. Crear entorno virtual python3 -m venv venv_deepdetect source venv_deepdetect/bin/activate # 2. Instalar PyTorch con soporte CUDA pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu121 # 3. Librerías de detección pip install opencv-python ffmpeg-python tqdm \ transformers==4.41.0 \ diffusers==0.27.0 \ facenet-pytorch==2.5.2 \ numpy==1.26.4 3.2. Descargar los modelos pre‑entrenados # Modelo de detección de artefactos de compresión (DeepDetect) git clone https://github.com/DeepDetectAI/deepdetect.git cd deepdetect git checkout v2.1 # versión optimizada para vídeo python -m pip install -e . # Modelo de análisis de parpadeo (BlinkCheck) basado en MediaPipe git clone https://github.com/google/mediapipe.git cd mediapipe python -m pip install . 3.3. Código de detección (ejemplo completo) import cv2, ffmpeg, numpy as np, torch from deepdetect import DeepDetectModel from mediapipe import solutions as mp_solutions from tqdm import tqdm # Cargar modelos dd_model = DeepDetectModel('weights/compression_artifacts.pt') blink = mp_solutions.face_mesh.FaceMesh( static_image_mode=False, max_num_faces=1, refine_landmarks=True, min_detection_confidence=0.5, min_tracking_confidence=0.5, ) def analizar_video(ruta): """Devuelve un puntaje de autenticidad entre 0 (fake) y 1 (real).""" # 1️⃣ Extraer frames (1 fps para rapidez) probe = ffmpeg.probe(ruta) fps = eval(probe['streams'][0]['r_frame_rate']) total_frames = int(probe['streams'][0]['duration'] * fps) step = int(fps) # 1 frame por segundo scores = [] for i in tqdm(range(0, total_frames, step)): out, _ = ( ffmpeg.input(ruta, ss=i/fps) .filter('scale', 640, -1) .output('pipe:', vframes=1, format='image2', vcodec='png') .run(capture_stdout=True, capture_stderr=True) ) img = cv2.imdecode(np.frombuffer(out, np.uint8), cv2.IMREAD_COLOR) # 2️⃣ Artefactos de compresión comp_score = dd_model.predict(img) # 3️⃣ Parpadeo anómalo results = blink.process(cv2.cvtColor(img, cv2.COLOR_BGR_RGB)) blink_score = 0.0 if results.multi_face_landmarks: # cálculo muy simplificado: distancia entre párpados landmarks = results.multi_face_landmarks[0].landmark eye_dist = np.linalg.norm( np.array([landmarks[159].x, landmarks[159].y]) - np.array([landmarks[145].x, landmarks[145].y]) ) blink_score = 1.0 - np.clip(eye_dist * 30, 0, 1) # 1 = normal # 4️⃣ Combinar final = 0.6 * comp_score + 0.4 * blink_score scores.append(final) return float(np.mean(scores)) if __name__ == '__main__': url = 'https://example.com/video.mp4' # o ruta local puntaje = analizar_video(url) print(f'🔎 Puntaje de autenticidad: {puntaje:.2f}') if puntaje < 0.5: print('⚠️ Probable deepfake') else: print('✅ Parece genuino') > Cómo funciona > 1. Compresión – El modelo DeepDetect reconoce patrones de cuantización típicos de los algoritmos de exportación de vídeo sintético. > 2. Parpadeo – Los deepfakes basados en GAN suelen generar parpadeos irregulares; MediaPipe mide la apertura de los párpados y detecta anomalías. > 3. Ponderación – Se combinan ambos indicadores (60 % compresión, 40 % parpadeo) para obtener un puntaje global. 3.4. Integración rápida en una página web Si prefieres un widget HTML/JS que cualquiera pueda usar sin instalar nada, copia este bloque en tu sitio y enlaza el script deepdetect-widget.js (disponible en el repositorio del artículo). <div id="deepfake-checker" style="max-width:480px;margin:auto;"></div> <script src="https://cdn.jsdelivr.net/gh/tuusuario/deepdetect-widget@v1.0/deepdetect-widget.js"></script> <script> DeepFakeWidget.init({ container: '#deepfake-checker', apiEndpoint: 'https://api.tuservidor.com/analyze', // tu backend con el script Python anterior language: 'es' }); </script> El widget permite arrastrar un archivo o pegar una URL y muestra el puntaje en segundos. --- 4. Checklist rápido de verificación (para periodistas y ciudadanos) --- Herramienta mencionada: Heroku
On-Device Computer Vision in React Native: Auto-Aligning Progress Photos with MediaPipe and Expo --- title: "On-Device Computer Vision in React Native: Auto-Aligning Progress Photos with MediaPipe and Expo" published: true description: "Derive a two-eye alignment transform, integrate on-device landmarks with Expo, and render progress timelapses without uploading photos." tags: reactnative, expo, computervision, mobile canonical_url: phasesapp.vercel.app --- A person can take thirty photos of the same face and still produce a timelapse that looks like camera shake. On Monday they stand closer to the mirror. On Tuesday they tilt their phone. By Friday they frame their eyes near the top of the image. For fitness, skincare, and haircut timelines, viewers need to see the change in the person without chasing their face around the screen. In Phases, the useful visual reference is the eye line: keep both eyes at the same output coordinates across the sequence. This article develops the React Native and Expo architecture for that workflow: detect landmarks on the device, compute a two-point transform, retain originals in local storage, and render a reel without a rendering server. Implementation scope: the public Phases site currently describes a browser app that users can add to their home screen. It also describes consent-based analytics. The native Expo architecture and zero-telemetry policy below describe the reference design for this article, rather than a verified description of that deployed browser build. The project link is Ishannaik/phases; the implementation examples below stand on their own. Separate the photograph from its alignment A capture guide helps the user repeat their pose, but it cannot guarantee matching camera distance or roll. I would keep the original photo and store its alignment metadata beside it. That lets the user change the reel's framing without reshooting or accumulating JPEG compression from repeated edits. flowchart TD A[Camera capture] --> B[Normalize orientation and mirroring] B --> C[On-device facial landmark detection] C --> D{Usable eye anchors?} D -->|Yes| E[Compute affine matrix] D -->|No| F[Ask user to retake or mark anchors] B --> G[Original photo in app filesystem] E --> H[SQLite metadata and transform] G --> I[Local Canvas or WebGL composite] H --> I I --> J[On-device video encoder] J --> K[Timelapse reel on device] G -. Optional user-enabled backup .-> L[Cloud backup] The engineer should treat capture, detection, and rendering as separate steps. After a successful detection, the renderer needs the photo URI and six transform coefficients. It does not need another inference pass each time the user plays the timeline. For a native app identified as io.phases.app, use its app-private filesystem for photos and SQLite for timestamps, dimensions, eye coordinates, and alignment settings. io.phases.app is an application identifier, not a portable filesystem path. Android and iOS choose the sandbox's physical location. With Expo, expo-file-system provides document storage, and expo-sqlite provides a persistent database. Store large image files in the filesystem rather than putting base64 photo strings into database rows. Choose eye anchors your detector can supply With MediaPipe, you can derive eye anchors from a face mesh. The older Face Mesh API offers iris refinement through refine_landmarks; the current Face Landmarker task provides a denser landmark result. Google documents 478 landmarks for its current model bundle. !MediaPipe canonical face mesh with landmark indices and eye-region topology Google's canonical face model UV visualization. This repository asset illustrates landmark topology, not a Phases screenshot or a detector result from this article. ML Kit can provide eye landmarks through its face detection API, but an eye landmark is not equivalent to a measured pupil center. If you choose ML Kit, name those anchors eye centers. If you need iris-center estimates, choose a detector configuration that supplies iris landmarks and document your index mapping against that model version. Even an iris center gives an estimate of the visible iris, not a clinical measurement of pupil position. For framing a selfie sequence, the distinction matters because eye movement can introduce jitter. A user looking left and then right can change the anchors without moving their head. A consistent gaze and capture guide help. Eye-contour centers offer another anchor convention when gaze changes dominate the result. I would run detection on each captured still image, with one intended face. A low-resolution preview can guide capture, but I would calculate the saved transform from the final image. Preview crops and final camera output often use different coordinate systems. Before measuring the eye vector: 1. Decode the photograph into a consistent upright orientation. 2. Apply the chosen mirror policy to both pixels and landmarks. 3. Convert detector coordinates into source-image pixels. 4. Preserve consistent eye identities across photographs. For a detector that returns normalized coordinates, multiply x by image width and y by image height. Measuring an angle in normalized coordinates distorts it whenever width and height differ. Two anchors determine rotation, scale, and translation Let the source eye anchors be: $$ p_1=(u_1,v_1),\qquad p_2=(u_2,v_2). $$ Choose fixed normalized output anchors: $$ q_1=(x_1,y_1),\qquad q_2=(x_2,y_2). $$ For horizontal eyes, choose $y_1=y_2$. In an output with width $W$ and height $H$, convert those targets to pixels: $$ Q_1=(Wx_1,Hy_1),\qquad Q_2=(Wx_2,Hy_2). $$ For example, on a 1080 by 1920 portrait canvas, (0.38, 0.35) and (0.62, 0.35) place the eyes at (410.4, 672) and (669.6, 672). Their separation is 259.2 pixels. These are illustrative framing choices, not measured Phases defaults. SOURCE PHOTO OUTPUT CANVAS p2 * Q1 *=============* Q2 / fixed eye axis / d p1 * tilted eye axis horizontal eye axis variable separation fixed separation translate + rotate + uniform scale Define the source and target eye vectors: $$ d=p_2-p_1,\qquad D=Q_2-Q_1. $$ Then calculate: $$ s=\frac{\lVert D\rVert}{\lVert d\rVert},\qquad \theta=\operatorname{atan2}(D_y,D_x)-\operatorname{atan2}(d_y,d_x). $$ Use the source midpoint $m=(p_1+p_2)/2$ and target midpoint $M=(Q_1+Q_2)/2$. For any source pixel $p$: $$ T(p)=sR(\theta)(p-m)+M, $$ where: $$ R(\theta)= \begin{bmatrix} \cos\theta & -\sin\theta\ \sin\theta & \cos\theta \end{bmatrix}. $$ You can write this as a homogeneous affine matrix: $$ \begin{bmatrix} x'\y'\1 \end{bmatrix} = \begin{bmatrix} a&c&e\b&d&f\0&0&1 \end{bmatrix} \begin{bmatrix} x\y\1 \end{bmatrix}. $$ Two point pairs determine this similarity transform, which uses uniform scale, rotation, and translation. They do not determine an arbitrary six-parameter affine transform with shear and independent axis scales. A similarity transform belongs to the affine family and preserves the face's proportions. For a horizontal target eye line, $\theta$ cancels the source eye angle. The coordinate system uses positive y downward, as Canvas does; calculate both angles in that same system. Calculate the Canvas coefficients in TypeScript The following reference implementation accepts source anchors in pixels and target anchors in normalized output coordinates. It returns the six values in the order that Canvas expects for setTransform(a, b, c, d, e, f). type Point = Readonly<{ x: number; y: number }>; type Affine = [number, number, number, number, number, number]; function eyeAlignment( source1: Point, source2: Point, target1: Point, target2: Point, width: number, height: number, ): Affine { const values = [ source1.x, source1.y, source2.x, source2.y, target1.x, target1.y, target2.x, target2.y, width, height, ]; if (!values.every(Number.isFinite) || width <= 0 || height <= 0) { throw new Error("Alignment needs finite coordinates and positive dimensions"); } if ([target1.x, target1.y, target2.x, target2.y] .some(value => value < 0 || value > 1)) { throw new Error("Target anchors must lie within normalized output bounds"); } const sx = source2.x - source1.x; const sy = source2.y - source1.y; const tx = (target2.x - target1.x) * width; const ty = (target2.y - target1.y) * height; const sourceDistance = Math.hypot(sx, sy); const targetDistance = Math.hypot(tx, ty); if (sourceDistance < 1 || targetDistance < 1) { throw new Error("Eye anchors must be distinct"); } const scale = targetDistance / sourceDistance; const angle = Math.atan2(ty, tx) - Math.atan2(sy, sx); const a = scale * Math.cos(angle); const b = scale * Math.sin(angle); const c = -b; const d = a; const mx = (source1.x + source2.x) / 2; const my = (source1.y + source2.y) / 2; const qx = (target1.x + target2.x) * width / 2; const qy = (target1.y + target2.y) * height / 2; const e = qx - a * mx - c * my; const f = qy - b * mx - d * my; return [a, b, c, d, e, f]; } function drawAlignedFrame( ctx: CanvasRenderingContext2D, image: CanvasImageSource, imageWidth: number, imageHeight: number, matrix: Affine, ): void { ctx.save(); try { ctx.setTransform(1, 0, 0, 1, 0, 0); ctx.clearRect(0, 0, ctx.canvas.width, ctx.canvas.height); ctx.fillStyle = "#000"; ctx.fillRect(0, 0, ctx.canvas.width, ctx.canvas.height); ctx.setTransform(...matrix); ctx.drawImage(image, 0, 0, imageWidth, imageHeight); } finally { ctx.restore(); } } Use imageWidth and imageHeight from the same oriented image coordinate system as the source anchors. The code assumes the caller owns the canvas, with no inherited clipping region. The one-pixel check prevents division by a degenerate eye vector. Production acceptance needs a stronger, image-relative criterion: a two-pixel eye separation on a large image can still produce an unusable scale. Check confidence, face size, and acceptable scale before saving alignment metadata. To check the matrix mathematically, apply it to each source anchor. You should recover the corresponding target pixel coordinate: a*u1 + c*v1 + e = W*x1 b*u1 + d*v1 + f = H*y1 a*u2 + c*v2 + e = W*x2 b*u2 + d*v2 + f = H*y2 These equations provide useful unit-test assertions with a floating-point tolerance. They describe invariants, not test results from a running Phases build. React Native needs a rendering boundary React Native does not expose a browser DOM Canvas as a built-in drawing surface. The CanvasRenderingContext2D example belongs in a browser or a WebView. In a native renderer, adapt the same coefficients to that renderer's matrix convention. You can use a WebView-hosted Canvas for a shared web rendering path, or use a native GPU drawing surface. With WebGL, upload a photo texture and apply the transform while drawing a quad. Account for the renderer's origin, texture orientation, and matrix layout; copying Canvas coefficients into an unrelated matrix API can invert or transpose the result. For MediaPipe or ML Kit integrations that require native code outside Expo Go, create an Expo development build. Installing a JavaScript wrapper cannot add its native implementation to the Expo Go binary. Expo documents this boundary in its development build guidance. Keep decoded image data near the detector and renderer. Passing a full-resolution base64 image through repeated JavaScript bridge messages adds allocation and copying. Transfer a URI or native image handle where the integration supports it. Render frames locally, then encode locally Drawing aligned photos on a canvas gives you frames. To export a video file, you also need an encoder and a container writer. In a native implementation, connect the rendering surface to an on-device encoder. In a browser implementation, select a supported local recording or encoding API and handle device-specific codec availability. Do not assume a Canvas screenshot sequence equals an MP4 export. For an export with one photograph per time interval, calculate presentation timestamps from the intended playback cadence. Processing speed should not determine the video's timing. A slow decode must delay export work, not stretch one day into a longer frame. Memory makes sequential processing important. One 1080 by 1920 RGBA frame takes 8,294,400 bytes, about 7.9 MiB before renderer and decoder overhead. Holding 365 such frames would consume about 2.8 GiB in pixel buffers alone. Decode a bounded number of photos, render a frame, submit it to the encoder, and release resources as the encoder permits. You should also choose how to handle uncovered canvas areas. Rotation can expose triangles at the corners; scaling cannot recover pixels outside the original capture. Use a consistent background, crop, or capture margin. Avoid changing the crop per frame, since that reintroduces apparent zoom. Local storage and an optional backup path A useful local record links a photo URI to its capture time, oriented dimensions, source eye anchors, detector version, and alignment settings version. Retain the originals so you can recalculate transforms after changing the target eye positions. Coordinate filesystem writes with database commits. Write the image into durable document storage before committing a row that references it. If the database commit fails, remove the new file or reconcile it on the next launch. SQLite cannot roll back a separate filesystem write. The reference design uses a zero-telemetry policy: no analytics events, session replay, or automatic diagnostic uploads. Developers must apply that policy to crash-reporting SDKs and dependencies as well as application code. Local inference does not by itself guarantee zero network traffic. For offline detection on first use, package the model with the app. A model download on first launch introduces a network prerequisite even if subsequent inference stays local. Keep cloud backup outside the alignment and rendering path. A user who declines backup should still capture, align, preview, and export. If they enable backup, explain which originals and metadata you upload, request the provider's permissions, and make local deletion versus remote deletion explicit. App-private storage also does not imply encryption or exclusion from operating-system backups; configure those policies for the threat model you support. The public Phases FAQ describes optional Google Drive backup and analytics that remain off until consent. Readers evaluating the deployed app should use that published policy rather than the stricter native reference policy above. Eye alignment has a defined limit A two-eye transform corrects translation, camera roll, and apparent face scale. It cannot correct a person turning their head, pitching their chin, changing expression, or photographing themselves under a different light source. On a fitness timeline, the eyes also cannot guarantee matching shoulder position or body pose. I would keep those constraints visible in capture UX: guide the user toward the same gaze, pose, and lighting, then use alignment to absorb small framing errors. For a failed detection or an occluded eye, offer a retake or manual anchors. Do not invent a plausible transform and label it a successful alignment. Phases' core visual goal is concrete: put the eye axis in the same place across days. With consistent image coordinates, a four-degree-of-freedom transform, and a local rendering pipeline, you can achieve that framing without sending a user's progress photos to an inference or rendering service. References - Phases and its published FAQ - Phases project repository - MediaPipe Face Landmarker model and task documentation - MediaPipe legacy Face Mesh and iris refinement - ML Kit face detection concepts - Expo SQLite - Expo filesystem Written by Ishan Naik.