tatopenn-cell

[open-source] Dense Armor Library : low-level toolkit for Build your Project Hi guys, I’ve been dedicating myself to researching papers for a while now to get a complete and unique set to train the next generation of intelligent software. The idea comes from wanting to simplify backend work, feeding the AI a ready-made and professional set to power your own program, AI agent, or drone. I developed the product with AI assistance and went through every single file by hand to make sure it respected the papers. Where there wasn't a solution in the papers, I had fun finding it by having the problem explained to me literally, solving it with intuition, using AI to translate, and Colab to test every attempt. Obviously, the hardest part was removing the hallucinations from what the AI passed off to me as ready, and I hope in part, thanks to the papers and the method, that I succeeded. Enjoy everyone, and if you like it, don't forget a star! pypi: pip install dense-armor info: tatopenn-cell.github.io

Show HN: A 20-line kernel that fixes float32 variance under XLA

Released Dense-Armor v1.2.0: A JAX-native stream learning engine with Mondrian tree routing that processes data 50x faster and cuts memory nodes by 3x. Composable attention and continual learning and memory land together: 15 new public classes, two docs pages, two benchmark families, one EFDT benchmark with its declared limit, and codecov.yml link: github.com/tatopenn-cell/Dense-Armor/releases/tag… XLA #JAX #StreamLearning #OnlineLearning #MachineLearning #Python #OpenSource

Vieni a divertirti con noi nella sezione Drone I droni sono la prossima sezione Strumenti online che eseguono campione per campione sui dati di un flight controller, su un Raspberry Pi o un NVIDIA Jetson accanto. Questo problema divide il lavoro in piccoli punti. Scegline uno, dillo in un commento e apri una PR. Ogni punto ha una domanda a cui risponde, una fonte di dati e un modo per verificarlo, quindi un primo contributo rientra in un pomeriggio. Come prendere un punto: commenta "Prendo N", fork, lavora su un branch, apri una PR che menziona questo problema. Un punto per PR. Le domande nei commenti sono benvenute. Regole fondamentali per ogni punto \- Ogni numero su una pagina deriva da uno script del repository eseguito prima che la frase fosse scritta. \- I dati sintetici sono etichettati "sintetici" ovunque appaiano. \- "Testato su un Raspberry Pi / un Jetson" è scritto solo quando il comando è stato eseguito su quella scheda; altrimenti "scritto, non eseguito sulla scheda". \- Le fonti sono i documenti nominati di seguito, con numeri di pagina. \## Dati e adattatori 1. \*\*Lettore PX4 ULog.\*\* Restituisce un dizionario per campione (tempo, giroscopio, accelerometro, velocità GPS, uscite del motore, tensione della batteria) con \`pyulog\`. Verifica: un piccolo ULog scritto dai round di prova. 2. \*\*Lettore MAVLink tlog.\*\* Gli stessi campi con \`pymavlink\`. Fonte: l'indagine MAVLink, arXiv:1906.10641 (Koubaa et al., IEEE Access 2019). 3. \*\*Generatore di volo sintetico.\*\* Hover più un cerchio, rumore, un guasto a un campione specificato, etichettato. La singola fonte di dati per test e prove di funzionalità. 4. \*\*Caricatore ALFA.\*\* Carica il dataset di guasti degli aerei a ala fissa (arXiv:1907.06268, Keipour et al., IJRR 2021; data DOI 10.1184/R1/12707963) e i suoi tempi di guasto veri. Verifica: i 47 voli processati si caricano e il tempo di guasto di ciascuno viene letto. \## Pipeline (ciascuna risponde a una domanda) 5. \*\*Quale parte si è rotta e dopo quanti secondi?\*\* Residuo tra tasso comandato e misurato, un rilevatore CUSUM con \`reset\_on\_alarm=True\`, un punteggio Mahalanobis robusto. Metriche: ritardo di rilevamento ed episodi di allerta per ora di volo normale. Dati: ALFA (punto 4). Correlato: Garg e Fan, arXiv:2309.09108. 6. \*\*Il GPS dice la verità?\*\* Confronto online della velocità GPS con l'IMU integrato. Injecta una rampa di offset nei voli normali e dillo sulla pagina. Motivazione: SemperFi arXiv:2105.01860 e Kim et al. arXiv:2407.15003. 7. \*\*Una elica è danneggiata?\*\* Spettro online del giroscopio o accelerometro e una distanza delle energie della banda dal baseline sano. Correlato: Zhang et al., arXiv:2302.04410. 8. \*\*Quanti minuti di volo sono rimasti e quanto siamo sicuri?\*\* Adattamento online della tensione contro l'attuale e il tempo con un intervallo; il tempo rimanente come una distribuzione. Fonte per il modello energetico: Alyassi et al., arXiv:1703.10049. 9. \*\*La prossima mossa è sicura?\*\* Distanza a un geofence e a ostacoli dal lidar simulato in \`benchmarks/lidar\_sim.py\`, con gli strumenti di protezione già nella libreria. 10. \*\*Dov'è il pad di atterraggio e la telecamera si sta muovendo?\*\* \`GlobalMotion\`, \`MotionMask\` e \`FrameAnomaly\` su frame sintetici di un pad. Metriche: errore di pixel rispetto alla posizione impiantata e tempo per frame a 640x360. 11. \*\*Il link radio sta per fallire?\*\* Un rilevatore di drift sul tempo di inter-arrivo dei pacchetti e RSSI di uno stream MQTT sintetico. 12. \*\*Statico contro online.\*\* Gli stessi dati di guasto con un modello batch di scikit-learn e quello online; mediana di 20 semi, tempo per campione, memoria. La pagina segue \`docs/finance/static\_vs\_online.md\`. \## Schede 13. \*\*Pagina Raspberry Pi a Pixhawk.\*\* Cablaggio seriale, i parametri PX4 e le impostazioni del Pi, con ogni passaggio verificato rispetto alla pagina di documentazione PX4 (docs.px4.io\_computer/pixhawk\_rpi.html). Segna ogni passaggio che hai eseguito tu stesso. 14. \*\*Pagina delle modalità di alimentazione Jetson.\*\* Le modalità di \`nvpmodel\` di ciascun modulo Orin e il tempo per campione di una pipeline in ciascuna modalità, misurato sulla scheda che possiedi. 15. \*\*\`edge\_bench.py\`.\*\* Tempo per campione (mediana e intervallo interquartile), 99° percentile, memoria residente e temperatura CPU per ciascuna pipeline su una certa scheda, scritto in un file JSON denominato secondo l'host. Includi il modello della tua scheda nel nome del file. I risultati di una scheda che possiedi sono il contributo più prezioso qui. 16. \*\*Dockerfiles per arm64 e Jetson, \`install.sh\`, un'unità systemd e un bridge MQTT\*\* che pubblicano il punteggio e gli episodi di allerta. I nomi delle immagini di base provengono dalla documentazione del fornitore, con l'URL. \## Anche aperto 17. \*\*ArduPilot.\*\* Gli stessi adattatori contro un log di ArduPilot, con le differenze scritte. 18. \*\*Un log di volo proprio.\*\* Esegui una pipeline su un log del tuo drone e dicci cosa ha trovato, come commento o come pagina. 19. \*\*Hai notato qualcosa nel piano?\*\* Una pipeline mancante, un documento sorgente migliore, una frase sbagliata. Apri un commento e diventa un punto numerato. Un nuovo contributore può iniziare con il punto 3, 13 o 18. La cartella con le istruzioni, le regole e i documenti è tenuta dal manutentore; chiedi qui se vuoi una copia. \## Documenti da leggere (uno per area) \- \*\*MAVLink, il protocollo a cui si riferiscono ogni punto 1-2 e 16:\*\* A. Koubaa et al., "Micro Air Vehicle Link (MAVLink) in a Nutshell: A Survey", IEEE Access 2019, arXiv:1906.10641. \- \*\*Dati di guasto e metriche (punti 4-5):\*\* A. Keipour, M. Mousaei, S. Scherer, "ALFA: A Dataset for UAV Fault and Anomaly Detection", IJRR 2021, arXiv:1907.06268. \- \*\*Guasti agli attuatori (punto 5):\*\* K. Garg, C. Fan, "Rilevamento e identificazione dei guasti basati su rete neurale per quadrotori utilizzando simmetria dinamica", arXiv:2309.09108. \- \*\*Attacchi GPS e per stimatore di stato (punto 6):\*\* H. Sathaye et al., "SemperFi: Un ricevitore GPS che elimina le falsificazioni per UAV", arXiv:2105.01860; K. H. Kim et al., "Prevedibile per progettazione, vulnerabile per natura: conseguenze di sicurezza dell'apprendimento negli stimatori di stato UAV", arXiv:2407.15003. \- \*\*Guasti all'elica (punto 7):\*\* W. Zhang et al., "Diagnosi dei guasti UAV dalla simulazione alla realtà con deep learning", arXiv:2302.04410. \- \*\*Batteria ed energia (punto 8):\*\* R. Alyassi et al., "Ricarica autonoma e pianificazione delle missioni di volo per droni autonomi a batteria", IEEE T-ASE 2023, arXiv:1703.10049. \- \*\*Schede (punti 13-16):\*\* Documentazione PX4, "Computer Compagni", docs.px4.io\_computer/ e "Raspberry Pi a Pixhawk", docs.px4.io\_computer/pixhawk\_rpi.html; modalità di alimentazione NVIDIA Jetson Orin, docs.nvidia.com. Conosci un documento migliore per una di queste aree? Aggiungilo in un commento con il numero arXiv. link : [ Dense-Evolution-Discovery#266 ](Dense-Evolution-Discovery#266)

Naive Bayes: algorithmic variants for streaming classification This page explains four simple, fast classifiers that learn by counting how often features and categories appear together. They update in a few microseconds, adapt to new data one sample at a time, and are designed specifically for text, events, and categorical logs rather than continuous numerical data. ## 1. What they are Imagine you want to classify a document, or a stream of events, or the contact state of a robot foot. You cannot see the underlying cause; you only see a handful of signals, and you want to guess the class each time a new signal arrives. Naive Bayes answers with the simplest rule that works: assume the signals are independent once you know the class, count how often each one appears in each class, and pick the class whose counts explain the sample best. This is a naive assumption, and it is often false. Two words in a sentence are not independent; two joints of the same arm are not independent. Yet the classifier still works, because classification only needs the *sign* of the decision, not a precise probability estimate. McCallum and Nigam (1998) made the assumption concrete for text, and the four variants here follow their event models. ## 2. The four variants They differ in what a sample is and how a feature contributes. - **MultinomialNB**: a sample is a bag of counts. Each feature has a count, and the class-conditional probability of a feature is the fraction of its occurrences that fall in that class, Laplace-smoothed (McCallum and Nigam 1998, eq. 5). Good for word counts, event counts, anything where the number matters. - **BernoulliNB**: a sample is a set of presence flags. Each feature contributes either its probability of appearing (if present) or the complementary `1 - p` (if absent). McCallum and Nigam 1998, eq. 2. Good when you only care whether something is there. - **ComplementNB**: the class-conditional probabilities are estimated from the *other* classes, not the one being scored. Rennie, Shih, Teevan and Karger (2003), eq. 6. Designed to correct the bias of MultinomialNB on imbalanced streams. - **CategoricalNB**: a sample is a set of categorical values, not counts. Each feature has its own vocabulary of categories, and the class-conditional probability of a category is its count divided by the class's total. Good for labels like "color: red". ## 3. How the update works Every sample is a dict. `learn_one(x, y)` adds one observation to the class `y`: MultinomialNB counts[y][f] += x[f] class_total[y] += sum(x.values()) BernoulliNB present[y][f] += (1 if x[f] > 0 else 0) class_n[y] += 1 ComplementNB counts[y][f] += x[f] (the other classes see it) total[y] += sum(x.values()) CategoricalNB counts[y][f][x[f]] += 1 class_n[y] += 1 At prediction time, the class scores are sums of log-probabilities, and the answer is the argmax. All four use Laplace smoothing: every count starts with a small positive value, so a feature never seen before does not make a class impossible. The smoothing parameter is `alpha`, default 1.0 (the Laplace prior). For MultinomialNB, the class-conditional probability of feature `f` is McCallum and Nigam, eq. 5: theta(f | c) = (1 + sum_i N_if P(c | x_i)) / (|V| + sum_f sum_i N_if P(c | x_i)) In words: the count of feature `f` in class `c`, plus one, divided by the total count of every feature in class `c`, plus the size of the vocabulary. For BernoulliNB, the same idea but on presence flags (eq. 3): `(1 + present) / (2 + n)`, where the 2 is the size of the boolean vocabulary. ## 4. Smallest example Four documents, two features each. The first two are class `X`, the last two class `Y`: ```python from dense_armor.utility.bayes import MultinomialNB docs = [ ({"a": 2, "b": 1, "c": 0}, "X"), ({"a": 1, "b": 0, "c": 2}, "X"), ({"a": 0, "b": 2, "c": 1}, "Y"), ({"a": 0, "b": 3, "c": 1}, "Y"), ] nb = MultinomialNB(alpha=1.0) for x, y in docs: nb.learn_one(x, y) print(nb.predict_one({"a": 5, "b": 0, "c": 0})) print(nb.predict_one({"a": 0, "b": 5, "c": 0})) ``` ``` X Y ``` After training, the counts inside the classifier are exact. For class `X`: feature `a` appears 3 times, `b` once, `c` twice, total 6. For class `Y`: `a` never, `b` 5 times, `c` twice, total 7. A new sample with five `a`s and no `b` or `c` therefore matches `X` far better than `Y`, with a posterior probability of 0.999. A sample with five `b`s and nothing else does the reverse, 0.993 for `Y`. ## 5. The two event models, seen  The same four documents, read by the multinomial model (left) and the Bernoulli model (right). The multinomial bars are the fraction of each feature within a class: `b` is rare in `X` and common in `Y`, which is what makes the classifier useful. The Bernoulli bars are the probability that each feature *appears at all* in a document of that class; the difference between the two is smaller, because the Bernoulli model throws away the counts.  Same query, two models. The query has `a = 0`, `b = 1`, `c = 1`. The multinomial model only looks at the present features, so it ignores `a`. The Bernoulli model additionally counts the absence of `a`, which is unusual for `X` and common for `Y`; that pushes the answer slightly further toward `Y`. This is the sense in which the Bernoulli model "counts absent words" and the multinomial model does not (McCallum and Nigam 1998, section 2). ## 6. Numbers On `DriftStream` sudden drift, 2000 samples, 4 continuous features, median of 5 seeds, p99 of one `learn_one` + `predict_one` over 3000 samples on one CPU thread: | classifier | accuracy | accuracy IQR | p99 | state | |---|---|---|---|---| | MultinomialNB | 0.089 | 0.214 | 0.034 ms | 421 B | | BernoulliNB | 0.089 | 0.214 | 0.035 ms | 380 B | | ComplementNB | 0.089 | 0.214 | 0.045 ms | 408 B | | CategoricalNB | 0.504 | 0.022 | 29.9 ms | 244 311 B | | OnlineGaussianNB (existing) | 0.996 | 0.030 | 0.087 ms | -- | The three count-based classifiers score 0.089 on this stream, **below** the chance level of 0.50. The reason is visible in the number of `None` returns: on the median seed 1823 of 2000 samples `predict_proba_one` is empty and `predict_one` returns `None` (across the five seeds the count ranges from 1572 to 2000, and the median accuracy 0.089 corresponds to seed 2). The count-based update treats a feature value as a count, and a negative value (which `DriftStream` produces on most samples) is skipped as missing; the class has then seen no evidence and returns nothing. `None == y` is never true, so the accuracy collapses to the fraction of samples whose features all happen to be non-negative. This is not chance, it is the boundary of the family: count-based models assume the inputs are counts (text, events, contact flags), never real-valued signals. `CategoricalNB` handles the stream because it treats each distinct value as a category, and on a balanced binary stream that gives 0.504, which *is* the chance level; the price is that every continuous sample creates a new category, the map grows, and the p99 cost rises to 29.9 ms. On this stream the right tool is `OnlineGaussianNB` at 0.996 accuracy and 0.087 ms p99; the count-based family here is a text and event classifier, not a real-valued one. ## Details ### References - McCallum, A., Nigam, K. (1998). A comparison of event models for naive Bayes text classification. AAAI Workshop on Learning for Text Categorization. Eq. 2 (Bernoulli), eq. 3 (Bernoulli parameter estimate), eq. 5 (multinomial). - Rennie, J. D. M., Shih, L., Teevan, J., Karger, D. R. (2003). Tackling the poor assumptions of naive Bayes text classifiers. ICML. Eq. 6 (complement parameter estimate) and the normalisation step. ### Choices stated explicitly - **Complement sign.** Rennie et al. write the classification weight as `w_ci = log(theta_{~c i})` and select the class with the smallest score (`argmin`). This implementation uses the equivalent sign-flipped form `w_ci = -log(theta_{~c i})` with the standard `argmax`, so that `predict_one` and `predict_proba_one` behave like every other classifier in the library. The change is stated in the module docstring. - **Normalisation.** `ComplementNB(norm=True)` divides the weight vector of each class by the sum of its absolute values, which is the normalisation described after eq. 6 in the paper. Set `norm=False` to keep the raw log probabilities. - **Laplace default.** `alpha=1.0`, the Laplace prior. The same parameter appears in MultinomialNB, BernoulliNB, ComplementNB, and CategoricalNB. `alpha=0` is rejected: without smoothing a single unseen feature would make a class impossible. - **Class prior.** By default the class prior is the observed frequency. `class_prior` can be passed as a dict to override it. - **Feature keys.** `feature_keys=None` takes the sorted keys of the first sample seen. - **Non-negative counts.** MultinomialNB, BernoulliNB, and ComplementNB treat a negative value as a missing sample and count it in `n_missing`. CategoricalNB treats a `None` or a non-finite float as missing; any other value (a string, a boolean) is a valid category. ### Measured cost p99 of one `learn_one` plus one `predict_one` over 3000 samples, 4 features, one CPU thread: 0.034 ms (MultinomialNB), 0.035 ms (BernoulliNB), 0.045 ms (ComplementNB), 29.9 ms (CategoricalNB). `budget_s` is 0.001 s for the first three, 0.1 s for CategoricalNB. State size after 3000 samples: 421 B, 380 B, 408 B, 244 311 B. The three count-based classifiers are `O(c d)` in the state (one counter per class and feature). CategoricalNB is `O(c d v)` where `v` is the number of distinct values seen, which on continuous inputs grows with the number of samples; the declared `memory_class` is `"O(c d v)"`, honest about the growth. ### No pure form The four classes do not provide an `init_state` / `step` pair. The state is a dict of counters per class, and its shape changes when a new class or a new feature appears; it is not a fixed-size array and would not survive `jax.lax.scan`. `check_estimator` runs `check_step_is_pure` and `check_step_is_jittable` on them with no check skipped, and both pass. ### Development diary The three count-based classifiers score 0.089 on `DriftStream` because that stream has continuous features and negative values, which the count-based update skips as missing. This is not a bug: it is the boundary of the family. When the count-based model is the right tool (text, events, contact states), it is also the fastest of the five: 0.034 ms p99 per sample against 0.087 ms for `OnlineGaussianNB`, 2.5 times faster, and against 29.9 ms for `CategoricalNB`, three orders of magnitude faster. The benchmark uses the same stream for every classifier, so the numbers are comparable even when the domain is wrong; the wrong-domain result is stated in the table rather than hidden by a different stream.

Risk: sequential quantile sketch architectures for streaming value-at-risk estimation Every morning, a bank or investment fund needs an estimate of how much money it could lose if the market turns bad. Instead of loading decades of historical files into slow spreadsheets, this tool processes new market returns one day at a time in less than a millisecond, using compact mathematical buckets to calculate risk percentiles with 1% accuracy while flagging if overall market volatility is starting to rise. ## The scene Before daily market operations commence, corporate risk committees and capital allocation desks require an immediate, audited quantification of extreme tail exposure: exactly how much capital is at risk if market conditions degrade. Traditional legacy approaches rely on heavy retrospective batch queries, dragging decades of asset returns into static spreadsheets—a manual process that introduces operational latency and misses intra-day volatility shifts. To maintain institutional compliance and defensive capital positioning, risk metrics must be calculated continuously and incrementally, updating perimeter exposure boundaries in real time as new sessions settle. The online risk pipeline was benchmarked over **1,253 historical trading days** of the S&P 500 index (spanning 14 October 2021 to 13 October 2026). The system updates distribution parameters sample by sample, delivering refreshed risk percentiles at opening intervals without requiring expensive data history re-scans or historical file management. ## Core Capabilities | Tool | Strategic Function | Operational Impact | |---|---|---| | **DDSketch** | Bounded tail-risk tracking | Maintains a fully bounded summary of extreme asset distributions, calculating high-accuracy risk percentiles without storing raw historical datasets. | | **CUSUM** | Volatility drift surveillance | Monitors cumulative micro-movements in loss distributions, immediately flagging subtle upward shifts in volatility regimes. |  *Each bar is the loss of one day. The red line is the 99% Value-at-Risk at the end of the run: 14 days lose more.* ## 1. High-accuracy tail-risk estimation (DDSketch) Quantifying exposure boundaries within severe market corrections requires tracking the exact shape of distribution tails. Traditional frequentist methods demand complete historical transaction persistence, creating large database scaling challenges. DDSketch bypasses this constraint by mapping incoming asset distributions into a logically indexed matrix of relative-error logarithmic buckets. This gives a quantile estimate with a maximum 1% relative error bound while maintaining a completely static, constant-memory footprint. ## 2. Proactive regime shift identification (CUSUM) Value-at-Risk boundaries represent an isolated snapshot of historical distributions. If structural market conditions undergo a slow, sub-clinical degradation, single-sample checks fail to register the risk change until major capital drawdowns occur. This module utilizes a sequential cumulative sum framework, continuously aggregating marginal directional shifts over time. When this statistical accumulation diverges from nominal white noise, a regime alarm triggers. ## Execution Workflow For every newly settled asset session or intra-day rolling window, the streaming risk pipeline runs a linear evaluation protocol: 1. **Quantile Mapping:** The session return vector is fed into **DDSketch**, updating distribution statistics and emitting calibrated 99% and 99.9% VaR thresholds. 2. **Sequential Drift Assessment:** The same vector passes to **CUSUM**, which evaluates whether cumulative structural losses are shifting systematically relative to earlier baseline windows. 3. **Institutional Reporting Output:** The pipeline returns two risk limits and one warning flag, with no rescan of past data. ## Empirical Performance The processing pipeline was rigorously verified against 1,253 active trading sessions, running a single streaming call per operational session. | Metric | Value | |---|---| | VaR 99% | 2.93% | | VaR 99.9% | 4.28% | | Days with a loss above the final VaR 99% | 14 out of 1,253 (1.1%) | | Days with a loss above the final VaR 99.9% | 3 out of 1,253 (0.24%) | | Days above a 5% loss | 1 out of 1,253 (4 April 2025, −5.97%) | | Days with a volatility-drift alarm | 49 out of 1,253 (3.9%) | | Time per day | 98 microseconds (median, Kaggle CPU) | ### Strategic Business Takeaway The final VaR 99% is the 99th percentile of these same 1,253 days, so about 1% of them fall above it: **1.1% of trading days** (14 days), and **0.24%** (3 days) above the 99.9% line. This checks the sketch against its target; it is not a forecast of future losses. The median time is **98 microseconds per day**, with no daily sweep over past data. ## Target Enterprise Use Cases This streaming risk architecture is designed for institutional deployment: - **Automated Executive Risk Reporting:** Generates opening capital-at-risk statistics for portfolio committees without technical data-wrangling overhead. - **Intraday Stress Testing:** Can be called via `process_day` across a rolling window (e.g., 60-minute slices) to evaluate intra-day structural regime deterioration. - **Regulatory Limit Enforcement:** The `over_limit` flag can be used as an automatic stop in an execution desk. ## Details ### Implementation The pipeline is in `pipelines/finance/risk.py`. Primitives chained, each with its own page: - [DDSketch](../stats/sketches.md) - [CUSUM](../drift/cusum.md) ### References - Masson, C., Rim, J. E., Lee, H. K. (2019). *DDSketch.* PVLDB 12(12), 2195-2205. arXiv:1908.10693. - Reynolds, M. R. Jr., Stoumbos, Z. G. (1975). *A CUSUM for monitoring the process mean.* - Market data: Yahoo Finance, S&P 500 (\^GSPC), 14 October 2021 to 13 October 2026. *These pages describe measurements on historical data. Not a forecast, not financial advice.* Git hub: github.com/tatopenn-cell/Dense-Armor/blob/master/…