zstd

Welcome to the new game format ZSTD—so long, lizard, and hello ZebraS-Tailed lizarD meta/zstd With this new format, for example the game Control Resonant finally works on firmware versions 12.xx and 13.xx 🦎 +🦓 = 🐉 t.co

Data Lifecycle & Storage Tiering for Cost Efficiency - [Why storage spend quietly becomes your platform's largest recurring tax] - [Design lifecycle rules and tiering that actually lower bills] - [Choose compression, formats and partitioning to shrink I/O and storage] - [Measure savings, compute ROI, and accept predictable trade-offs] - [A practical, runnable playbook: lifecycle snippets, compaction jobs, and checklists] Storage costs compound — not in sudden, dramatic outages, but as a steady monthly tax that eats your analytics margin. You control that tax with disciplined data lifecycle policies, pragmatic storage tiering, and a data layout that minimizes scanned bytes. Many teams reach the same symptoms: monthly cloud storage bills creeping upward, dashboards showing terabytes scanned per query, hundreds of thousands (or millions) of tiny objects that blow up request and list costs, and surprise charges from restores or early-deletion penalties. S3 and other clouds provide robust lifecycle tools, but there are gotchas — for example, S3 Intelligent-Tiering excludes auto-tiering for objects smaller than 128 KB and carries per-object monitoring nuances . GCS lifecycle actions can lag and may take up to 24 hours to begin acting after you change rules . Azure applies minimum retention windows and early-deletion pro-rations that you must account for when tiering into archive . Why storage spend quietly becomes your platform's largest recurring tax Storage scales predictably with data retention, replication, and poor layout. A few structural realities make the bill grow faster than teams expect: - Every extra copy multiplies costs. Backups, snapshots, and raw+processed copies multiply bytes stored; each copy multiplies per-GB-month charges and per-object request footprints . - Small files increase operational overhead. Thousands of tiny objects create request, LIST, and metadata costs and slow planning phases in query engines — producing higher CPU time and longer query latency . - Tier misalignment and retention rules leak money. Moving objects to long-term archive without accounting for minimum storage durations leads to pro-rated charges; archives have cheaper per-GB rates but higher retrieval/rehydration costs and latency . - Blind ingestion keeps everything hot. Treating raw event streams as permanent first-class citizens without TTLs or tagging guarantees long-term spend. > Important: Archive tiers have minimum retention periods and rehydration costs; S3 Glacier Deep Archive commonly requires 180 days and Azure archive has 180 days as well — deleting earlier incurs pro-rated charges. Model those penalties into any tiering plan. Design lifecycle rules and tiering that actually lower bills Good lifecycle and tiering design reduces bill surface area while preserving business value. Think in rulesets, not one-offs. Core patterns that work in practice: - Age + tag + prefix rules. Combine age with tags or prefixes so you only tier/delete intended subsets (e.g., backups/ vs processed/). S3 lifecycle rules and filters support prefix and tag filters to scope actions. - Staged transitions. Use a staged path: Hot → Infrequent → Archive, with thresholds aligned to access patterns (30/90/365 days are common anchors). AWS, GCP, and Azure all support multi-step transitions and versioned-object transitions. - Intelligent vs explicit tiering. S3 Intelligent-Tiering automates moves based on access patterns but has monitoring semantics and object-eligibility details to account for; explicit transitions reduce surprise but require you to know access patterns. Choose based on predictability of access and per-object counts. - Versioned and noncurrent handling. Noncurrent versions inflate storage. Use lifecycle rules to transition or expire noncurrent versions after a retention window rather than keeping unlimited history. S3 lifecycle supports NoncurrentVersionTransition and NoncurrentVersionExpiration. Practical rule design checklist (strategy-level): - Tag candidate datasets by retention class (e.g., hot/nearline/archive/compliance). - For unknown access patterns, use short intelligent tiers or short monitoring windows; for known cold datasets, use aggressive explicit archiving. - Test lifecycle rules against a dev bucket and a small production subset — lifecycle changes can take time to propagate. GCS warns that changes can take up to 24 hours to take full effect. Example S3 lifecycle (JSON) { "Rules": [ { "ID": "analytics-tiering", "Filter": { "Prefix": "raw-events/" }, "Status": "Enabled", "Transitions": [ { "Days": 30, "StorageClass": "STANDARD_IA" }, { "Days": 90, "StorageClass": "GLACIER" } ], "Expiration": { "Days": 1825 } } ] } This pattern moves raw-events/ first to infrequent, then to Glacier, and expires after 5 years. Use precise scoping (prefix/tags) so unrelated objects aren’t swept away. Example GCS lifecycle (JSON) { "lifecycle": { "rule": [ { "action": { "type": "SetStorageClass", "storageClass": "COLDLINE" }, "condition": { "age": 365, "matchesStorageClass": ["STANDARD"] } } ] } } GCS supports SetStorageClass, Delete, and conditions like age, matchesPrefix, matchesSuffix — and will evaluate rules asynchronously. Example Azure lifecycle (JSON) { "rules": [ { "name": "archiveRule", "enabled": true, "type": "Lifecycle", "definition": { "filters": { "blobTypes": ["blockBlob"], "prefixMatch": ["archive/"] }, "actions": { "baseBlob": { "tierToArchive": { "daysAfterModificationGreaterThan": 90 } } } } } ] } Azure lifecycle supports tierToCool, tierToArchive, tierToCold and delete actions, plus run conditions; plan for rehydration latencies and early-deletion rules. Choose compression, formats and partitioning to shrink I/O and storage Storage tiering buys you dollars-per-gigabyte savings; data layout and compression shrink the denominator. - Use columnar formats for analytics. Parquet or ORC dramatically reduce bytes scanned compared with JSON/CSV by storing column pages and enabling predicate pushdown and column pruning. Parquet supports multiple compression codecs and page/row-group tuning. - Pick codecs to match access patterns. Snappy is fast for active data (low CPU cost, good decompression throughput). Zstandard (zstd) typically gives significantly better compression ratios at acceptable CPU cost and is now commonly supported by engines and by Parquet implementations — valuable for long-lived or infrequently-read data. Benchmarks and the Parquet spec show ZSTD as a supported codec with compelling ratios vs older codecs. Test on representative data to pick codec/level. - Target file sizes for efficient reads. Many engines (Athena, Spark/Delta) optimize for file sizes in the low hundreds of megabytes (commonly 128–512 MB or an adaptive target). Too-small files increase scheduling and request overhead; too-large files hurt parallelism and update granularity. Databricks gives guidance on delta.targetFileSize and auto-compaction behavior; Athena docs recommend aiming for ~128 MB splits for parallelism. - Partition sensibly (and sparingly). Partition by a low-cardinality, high-selectivity field that appears in the majority of queries (commonly date in year/month/day hierarchy). Avoid high-cardinality keys (e.g., user_id) as partition keys unless you use partition projection / partition indexing. Over-partitioning leads to too many tiny partitions and metadata overhead. - Sort / cluster to enable data skipping. Within files, sort (or ZORDER / clustering in Delta/Iceberg) on common filter columns to maximize min/max statistics and block skipping. Sorted files + column stats let query engines skip whole row groups. Example: write Parquet with zstd (PySpark) # set codec (confirm runtime support) spark.conf.set("spark.sql.parquet.compression.codec", "zstd") # or "snappy" (df.write .mode("append") .partitionBy("year", "month", "day") .parquet("s3://company-data/events/")) Confirm zstd support on your engine/runtime before adopting broadly — not all older runtimes support every codec. Compaction approach (coalesce small files per partition): from pyspark.sql import SparkSession spark = SparkSession.builder.getOrCreate() path = "s3://company-data/events/date=2025-01-01" df = spark.read.parquet(path) df.repartition(16).write.mode("overwrite").parquet(path) On managed Delta tables, prefer OPTIMIZE / ZORDER or the engine’s auto-compaction features instead of ad-hoc overwrite loops. Databricks and Delta provide built-in autotuning and OPTIMIZE primitives. Measure savings, compute ROI, and accept predictable trade-offs Optimization is a measurement problem. Build an ROI model that includes both avoided storage costs and the operational/latency trade-offs. Core measurement steps: 1. Inventory and baseline. Use provider tooling: S3 Inventory, S3 Storage Lens, GCS Storage Insights, or Azure Storage metrics to capture object counts, sizes, and access patterns by prefix/tag. Record: object count, total GB, monthly GET/PUT counts, and common query scan sizes. 2. Model transitions. For each candidate dataset, compute: - Current monthly storage = size_GB * price_per_GB_month (per-tier) - After-change storage = (size_GB * compression_gain) * price_per_GB_month_new - Add transition cost = PUT/COPY/lifecycle transition requests + any per-object monitoring fees (Intelligent-Tiering) + compaction compute cost. - Add expected retrieval costs if you expect restores. This algebra identifies break-even retention time for tier moves. 3. Account for minimum storage durations and request costs. Cloud providers impose minimum storage durations (e.g., Glacier 90/180 days; Azure archive 180 days) and charges for API operations. Include them in your ROI window. 4. Run a small pilot and observe real retrievals. Pilot a subset, monitor retrieval rates and restore latencies, and compare forecasted vs actual. Use that data to tune thresholds. 5. Track ongoing KPIs. Measure bytes_stored_by_tier, monthly_requests, avg_query_bytes_scanned, compute_seconds_for_compaction, and restore_events to prove savings. Simple ROI worksheet columns you can use: - Dataset | Current GB | Current tier unit $/GB | Expected compressed GB | Target tier $/GB | Transition cost (requests + compute) | Monthly saving | Months to break-even Operational trade-offs to surface: - Increased latency for archived data (hours to rehydrate vs milliseconds for hot). - Restore costs that may dwarf storage savings if restores are frequent. - Early deletion penalties if you mis-estimate retention and move data to archive then delete or move back too soon. - Compute cost of compaction (transient cluster/Glue jobs) vs savings from fewer objects and lower egress. Include those in your ROI model. A practical, runnable playbook: lifecycle snippets, compaction jobs, and checklists This is the tactical checklist I run when I inherit a high-cost data lake. Treat it as a short playbook you can run in a week. 1. Inventory (day 0–1) - Export per-prefix/object metrics using provider tools: S3 Inventory / Storage Lens, gcloud storage buckets describe --format=json, or Azure Storage Analytics. Capture sizes, object counts, last-modified times, and access counts. 2. Tagging pass (day 1–2) - Identify buckets/prefixes for hot, warm, cold, compliance and apply tags/labels. 3. Design rules (day 2) - For each tag/prefix, write lifecycle JSON (examples above). Use staged transitions and conservative windows at first (e.g., 60/180/365). 4. Pilot rollout (day 3–7) - Apply rules to a non-critical prefix and monitor for unexpected restores, errors, or rule misfires. GCS lifecycle changes may take up to 24 hours to propagate; plan accordingly. 5. Compact & compress (ongoing) - Schedule compaction jobs (Spark/Glue/Databricks) to coalesce small files weekly or after major writes. Use engine-native OPTIMIZE for Delta/Iceberg where available to avoid manual overwrite churn. 6. Measure and iterate (ongoing) - Compute monthly savings vs baseline, subtract compaction/transition costs, and iterate thresholds. Keep a running cost dashboard by prefix/tier. Operational quick-check checklist: - Cataloged datasets by tag? ✅ - Rules scoped by prefix & tag (not global)? ✅ - Pilot applied for at least 1 billing cycle? ✅ - Compaction job scheduled for high-ingest prefixes? ✅ - Monitoring dashboards: bytes_by_tier, restores, request_count? ✅ Example compact job (Spark job outline): # run as scheduled job on a worker cluster spark-submit --class com.company.CompactFiles \ --conf spark.sql.parquet.compression.codec=zstd \ compact_files.py --input s3://company-data/events/ --target-file-size 268435456 Example Databricks SQL optimize: -- reduce small files and improve skipping OPTIMIZE delta.`/mnt/delta/events` WHERE date >= '2025-01-01' ZORDER BY (user_id) Databricks documents autotuning and target file sizing primitives for Delta tables to automate much of this work. Sources Lifecycle configuration elements - Amazon S3 User Guide - Details on S3 lifecycle rule elements, filters (prefix/tags/size), Transition and Expiration actions, and noncurrent version handling. Amazon S3 Intelligent-Tiering Storage Class | AWS - Description of S3 Intelligent-Tiering access tiers, monitoring behavior, eligibility thresholds, and how objects move between tiers. Amazon S3 Pricing - Pricing components, minimum storage duration notes, request and retrieval charges, and examples for billing considerations referenced for ROI modeling. Object Lifecycle Management | Cloud Storage | Google Cloud - GCS lifecycle actions (SetStorageClass, Delete), rule conditions, examples, and operational notes (including the 24-hour propagation guidance). Access tiers for blob data - Azure Storage - Azure Blob access tiers (Hot/Cool/Cold/Archive), minimum retention durations, rehydration behavior, and early deletion penalties. Apache Parquet Format (spec / repo) - Parquet format documentation, supported compression codecs, page/block metadata, and format-level considerations for columnar storage and predicate pushdown. Configure Delta Lake to control data file size - Databricks Docs - Databricks guidance on delta.targetFileSize, auto-compaction/optimized writes, OPTIMIZE, and recommended target file sizing behavior. Top 10 Performance Tuning Tips for Amazon Athena | AWS Big Data Blog - Athena guidance covering partitioning, avoiding small files, compression advice, and split sizing recommendations. Zstandard (zstd) — GitHub - Official Zstandard implementation and benchmark references showing compression ratio and performance trade-offs compared to other codecs. Examples of S3 Lifecycle configurations - Amazon S3 User Guide - Concrete XML/JSON lifecycle examples for staged transitions, noncurrent version handling, and small-object transition exceptions. A focused lifecycle, conservative tiering windows, right-sized files, and measured compression choices will materially reduce your storage burn while keeping the data usable and reliable.

Webhook Payload Compression at Scale: Evaluating Brotli, Zstandard (zstd), and Gzip for High-Volume Ingress Most compression advice for APIs is about responses. Webhook gateways have the opposite problem. Providers push JSON into your infrastructure, often in bursts, and the request body is the part you pay to receive, buffer, verify and forward. This article looks at when compressing inbound webhook bodies is worth it, how Gzip, Brotli and Zstandard compare for that job, and how to decompress untrusted request bodies without opening a denial-of-service hole. It uses published benchmarks and vendor documentation, and says plainly where no hard numbers exist. 1. Is ingress compression worth it? Start with the cost model. On the major clouds, data coming in from the public internet is generally not what you are billed for, so compressing inbound webhooks will not by itself shrink a bill. Check your own provider’s pricing. The savings usually show up elsewhere: - Bytes moved through metered hops inside your network, such as NAT, private-link or transit-style gateways, and cross-zone traffic between the gateway and your workers. - Buffering pressure on the proxy, and queue or log storage if you persist raw bodies. - Provider-to-you transfer time for large payloads, which matters for retries and timeouts. Compression also moves CPU cost onto your gateway, which has to decompress every body. A sender that compresses can also shift cost onto you if you are careless about limits (see section 5). There is also an adoption question. Mainstream providers document their webhooks as plain JSON. GitHub, for example, caps payloads at 25 MB and signs the request body with HMAC-SHA256 in X-Hub-Signature-256 (GitHub docs). Compressed webhooks do exist, but they are a feature of specific senders. Stream Chat documents Content-Encoding: gzip on its HTTP webhooks (Stream docs). QuickNode Streams may send gzip bodies and notes that signature verification has to account for them (QuickNode guide). Crawlbase’s Enterprise Crawler delivers its callbacks gzip-compressed (Crawlbase docs). I could not find a major webhook provider that documents sending zstd or br request bodies. In practice, gzip is what you will receive from third parties today. Zstandard is most realistic for traffic between services you control, or from partners who agree to it. 2. How request-body compression works A webhook is an HTTP POST. If the sender compresses the body, it declares the algorithm in the Content-Encoding request header, for example Content-Encoding: zstd. The opposite direction is optional. A server can advertise which request content-codings it accepts by sending Accept-Encoding in its responses, but most webhook senders will not act on that. Treat it as documentation for integrators rather than automatic negotiation. In Envoy, the decompressor filter has an advertise_accept_encoding option for the request direction that does this advertising (API reference). JSON is a good fit for general-purpose compressors. It is text with repeated keys, repeated punctuation, and recurring value shapes such as timestamps and IDs. Real ratios depend on your payloads, which is why section 4 shows how to measure them. 3. Gzip, Brotli and Zstandard compared | | Gzip | Brotli | Zstandard | | --- | --- | --- | --- | | Content-coding token | gzip | br | zstd | | Specification | RFC 1952 (wrapping DEFLATE) | RFC 7932 | RFC 8878; HTTP window limit in RFC 9659 | | Entropy coding | Huffman | Huffman with context modelling | Huffman and finite state entropy (FSE/ANS) | | Built-in dictionary | No | Yes, about 120 KB static dictionary | No, but supports trained custom dictionaries | | Typical role | Universal fallback | Strong ratios, mostly used for web responses | Fast in both directions, tunable levels | Gzip. It is supported nearly everywhere, which is its main strength. Its DEFLATE window is 32 KB, so decompression memory per stream is small and predictable. Brotli. It ships with a static dictionary of roughly 120 KB, made of common words and snippets from natural languages and web content, which helps on small text. It is excellent at high levels for pre-compressed static assets. For on-the-fly compression, the high levels are expensive, which makes them a poor choice for a sender compressing each webhook as it is produced. Zstandard. It was designed for real-time use, with a wide range of levels and fast decompression. It also supports dictionaries trained on sample data, which must be available on both ends. For HTTP, RFC 9659 requires decoders to support windows up to 8 MB and forbids encoders from producing frames that need more (RFC 9659). Budget for that when sizing memory under concurrency. What published benchmarks show I have no trustworthy JSON-webhook benchmark to cite, and an invented one would be worse than none. The most useful public reference is the Zstandard project’s own comparison, run with lzbench on the Silesia corpus (a mixed corpus, not JSON) (facebook/zstd): | Compressor | Ratio | Compression | Decompression | | --- | --- | --- | --- | | zstd 1.5.6, level 1 | 2.887 | 510 MB/s | 1580 MB/s | | zlib 1.2.11, level 1 | 2.743 | 95 MB/s | 400 MB/s | | brotli 1.0.9, level 0 | 2.702 | 395 MB/s | 430 MB/s | At comparable fast settings, zstd compresses about five times faster than zlib and decompresses nearly four times faster, with a slightly better ratio. The project also notes that decompression speed stays roughly constant as you raise the compression level, which is why a sender can spend more CPU on ratio without slowing your gateway. Two cautions apply. These are corpus-wide numbers at the fastest levels, so absolute figures will differ on your hardware and your JSON. Library versions and bindings also matter, and Go implementations will not match the C reference. 4. Measure your own payloads Before changing anything, benchmark samples of your real traffic. The reference tools include benchmarking modes: # Collect a few hundred representative webhook bodies into ./samples/ # Zstandard: benchmark levels 1-5 on one file or a concatenation zstd -b1 -e5 samples/sample.json # Compare against gzip and brotli at the levels a sender would plausibly use for lvl in 1 6; do gzip -c -$lvl samples/sample.json | wc -c; done for q in 4 5; do brotli -c -q $q samples/sample.json | wc -c; done Measure per-payload sizes, not one large concatenation, because per-request compression of small bodies behaves differently from compressing a big file. If payloads are small, also try a trained zstd dictionary: zstd --train samples/* -o webhook.dict zstd -D webhook.dict -3 samples/sample.json A dictionary has to be shared out of band and versioned carefully, since a sender and receiver with different dictionaries cannot interoperate. It is a good fit for partner or internal traffic, and a poor one for public third-party providers. 5. Security: decompression bombs and memory limits Accepting compressed request bodies means accepting attacker-controlled expansion. A tiny body can decompress into a very large one. This is a recognised vulnerability class. Envoy shipped a fix for a decompressor zip-bomb issue (CVE-2022-29225) that capped the maximum inflation size (Envoy 1.22.1 release notes). Defences that matter, in rough order of importance: 1. Cap the decompressed size. This is the primary control. Enforce it while streaming, and fail when it is exceeded rather than silently truncating. 2. Cap the compressed size too. Limit what you will read off the wire before any decoder runs. 3. Bound decoder memory. Zstd’s window can be up to 8 MB per stream under the HTTP rules, so many concurrent streams add up. 4. Use an expansion-ratio limit as a secondary signal only. JSON can legitimately compress by large factors, so a fixed threshold can reject valid traffic. Choose one from your observed traffic and alert before you enforce. Envoy’s gzip decompressor exposes max_inflate_ratio for this purpose. 5. Never call io.ReadAll on a decoder unless it is already wrapped in a hard cap. 6. Reject unknown encodings. RFC 9110 lets a server answer with 415 Unsupported Media Type when it does not support a request’s Content-Encoding. Use 413 when a limit is exceeded. A bounded decompression middleware in Go This sketch uses the standard library’s compress/gzip, github.com/andybalholm/brotli and github.com/klauspost/compress/zstd. A detail worth noting: the klauspost zstd Decoder.Close() returns nothing and must be called to release resources, so it cannot be treated as an io.Closer directly. It needs an adapter. package ingress import ( "compress/gzip" "errors" "io" "net/http" "strings" "github.com/andybalholm/brotli" "github.com/klauspost/compress/zstd" ) const ( maxCompressedBytes int64 = 1 << 20 // read at most 1 MiB off the wire maxDecompressedBytes int64 = 10 << 20 // hand at most 10 MiB to handlers maxZstdWindowBytes uint64 = 8 << 20 // RFC 9659 window limit for HTTP ) // ErrDecompressedTooLarge is returned by Read when the decoded body exceeds the cap. var ErrDecompressedTooLarge = errors.New("decompressed body exceeds limit") // cappedReader fails loudly (unlike io.LimitReader) when the cap is exceeded. type cappedReader struct { r io.Reader remaining int64 } func (c *cappedReader) Read(p []byte) (int, error) { if c.remaining < 0 { return 0, ErrDecompressedTooLarge } // Allow one byte past the cap so overflow can be detected. if int64(len(p)) > c.remaining+1 { p = p[:c.remaining+1] } n, err := c.r.Read(p) c.remaining -= int64(n) if c.remaining < 0 { return 0, ErrDecompressedTooLarge } return n, err } type decodedBody struct { io.Reader closers []func() error } func (d *decodedBody) Close() error { var first error for _, c := range d.closers { if err := c(); err != nil && first == nil { first = err } } return first } // Decompress transparently decodes gzip, br and zstd request bodies with hard limits. func Decompress(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { enc := strings.ToLower(strings.TrimSpace(r.Header.Get("Content-Encoding"))) if enc == "" || enc == "identity" { r.Body = http.MaxBytesReader(w, r.Body, maxDecompressedBytes) next.ServeHTTP(w, r) return } compressed := http.MaxBytesReader(w, r.Body, maxCompressedBytes) var ( rd io.Reader closers []func() error ) switch enc { case "gzip": zr, err := gzip.NewReader(compressed) if err != nil { http.Error(w, "invalid gzip body", http.StatusBadRequest) return } rd, closers = zr, append(closers, zr.Close) case "br": rd = brotli.NewReader(compressed) case "zstd": zr, err := zstd.NewReader(compressed, zstd.WithDecoderConcurrency(1), zstd.WithDecoderMaxMemory(maxZstdWindowBytes), ) if err != nil { http.Error(w, "invalid zstd body", http.StatusBadRequest) return } rd = zr closers = append(closers, func() error { zr.Close(); return nil }) default: http.Error(w, "unsupported Content-Encoding", http.StatusUnsupportedMediaType) return } r.Body = &decodedBody{ Reader: &cappedReader{r: rd, remaining: maxDecompressedBytes}, closers: closers, } r.Header.Del("Content-Encoding") r.Header.Del("Content-Length") r.ContentLength = -1 next.ServeHTTP(w, r) }) } Downstream handlers should map ErrDecompressedTooLarge and *http.MaxBytesError to 413 Payload Too Large. A truncated or corrupt stream surfaces as a read error, which should become a 400. Signature verification Whether to verify the signature before or after decompression depends on what the provider signed, so read each provider’s documentation. GitHub documents its HMAC as computed over the request body (GitHub docs), and GitHub sends plain JSON. For a sender that compresses, QuickNode’s guidance shows verification being done on the decoded JSON string. If the provider signs the bytes on the wire, hash the compressed stream with io.TeeReader as you read it. If it signs the decoded payload, hash the output of the capped reader. In both cases, hash only bytes that have already passed your limits, and compare signatures in constant time. The middleware above discards the raw compressed bytes, so it suits only the second case unless you add the tee. Don’t sniff magic bytes It is tempting to detect compression by inspecting leading bytes when a sender forgets the header. Gzip starts with 1F 8B and zstd frames start with 28 B5 2F FD, but Brotli has no fixed signature. Guessing also makes your parser’s behaviour depend on attacker-controlled content. A sturdier policy is to trust the declared Content-Encoding, reject mismatches with 400, and tell integrators to fix their sender. 6. Offloading decompression to Envoy Envoy’s HTTP decompressor filter supports gzip, Brotli and Zstandard (Envoy docs). It can be enabled separately for requests and responses, and each filter instance is configured with one decompressor library. To accept several encodings, you add one filter instance per library. The docs also say a buffer filter must come after the decompressor for requests. The following is a structural sketch of an HTTP filter chain, not a complete bootstrap (listener, TLS and cluster definitions are omitted). Check field names against the Envoy version you run. http_filters: - name: gzip_request_decompressor typed_config: "@type": type.googleapis.com/envoy.extensions.filters.http.decompressor.v3.Decompressor decompressor_library: name: gzip typed_config: "@type": type.googleapis.com/envoy.extensions.compression.gzip.decompressor.v3.Gzip max_inflate_ratio: 50 # tune from observed traffic request_direction_config: common_config: enabled: default_value: true runtime_key: gzip_request_decompression_enabled advertise_accept_encoding: true response_direction_config: common_config: enabled: default_value: false runtime_key: gzip_response_decompression_enabled - name: zstd_request_decompressor typed_config: "@type": type.googleapis.com/envoy.extensions.filters.http.decompressor.v3.Decompressor decompressor_library: name: zstd typed_config: "@type": type.googleapis.com/envoy.extensions.compression.zstd.decompressor.v3.Zstd request_direction_config: common_config: enabled: default_value: true runtime_key: zstd_request_decompression_enabled advertise_accept_encoding: false response_direction_config: common_config: enabled: default_value: false runtime_key: zstd_response_decompression_enabled - name: envoy.filters.http.buffer typed_config: "@type": type.googleapis.com/envoy.extensions.filters.http.buffer.v3.Buffer max_request_bytes: 10485760 - name: envoy.filters.http.router typed_config: "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router Keep Envoy current. The zip-bomb fix mentioned above landed in patch releases back in 2022, and the inflation protection is on by default. If you terminate in NGINX, check whether your build supports request-body decompression for the encodings you need. Do not assume it from response-side compression modules. 7. Recommendations 1. Support gzip first. It is the only encoding you are likely to receive from third-party providers today. 2. Add zstd for traffic you control. It combines fast decompression with good ratios, and you can pair it with a dictionary for partner integrations. Remember the 8 MB window limit when planning memory. 3. Avoid expensive Brotli levels for senders compressing in real time. Brotli is a better fit for static assets than for per-request compression. 4. Cap everything. Compressed size, decompressed size and decoder memory should all be bounded, and failures should be explicit 413 or 400 responses. 5. Treat ratio limits as tuned guardrails, not universal constants. 6. Verify signatures over the bytes the provider actually signed, and only after limits are enforced. 7. Measure before rolling out. Use samples of your real payloads, and track decompression CPU, memory per connection and rejection rates once live. Sources - GitHub: Webhook events and payloads - Zstandard project README and benchmarks - RFC 9659: Window Sizing for Zstandard Content Encoding - Envoy: Decompressor filter - Envoy 1.22.1 release notes (CVE-2022-29225) - klauspost/compress zstd package documentation - Stream Chat webhooks overview - QuickNode: Validating incoming Streams webhook messages - Crawlbase: Webhook receiving --- Originally published at instawebhook.com

@ClaudeDevs For your next improvement facebook/zstd

I Saved 65.9% on the Data I Trained On, 53.5% on Unseen Data Compressing small records one at a time does not work. A 900-byte JSON object gives the compressor no history to learn from; the file ends before it has even memorised the field names. zstd's trained dictionary feature exists precisely to fill that hole: it extracts common patterns from your corpus and hands the compressor a pre-loaded history that says "you already know these". Tonight I put that promise to the test against my own blog's feed.json output. The first result looked wonderful: the dictionary shrank per-record output by 65.9%. Then I tried the same dictionary on records it had never seen during training. The gain fell to 53.5%. Those 12.4 points were not the dictionary's doing — they were my measurement error. And that same error sits in most articles written about zstd dictionaries, because everyone measures on the corpus they trained on. What follows is that whole evening: the commands, the raw output, and a concrete threshold for where to stop growing the dictionary. One machine, one zstd binary, a corpus anyone can download — repeat it yourself if you like. The corpus: my own blog feed I did not want to invent synthetic records for this experiment. I wanted real production data that anyone could also fetch. The blog's JSON Feed output fits exactly: 1,346 posts when I pulled it on the evening of 17 October 2026, each one a separate record. The feed is live and growing, so if you repeat this a few weeks from now the record count will be higher and the absolute figures will not match — the ratios will. curl -s https://mustafaerbay.com.tr/feed.json -o feed.json python3 - <<'EOF' import json, os d = json.load(open('feed.json')) items = sorted(d['items'], key=lambda x: (x['date_published'], x['id'])) for name, arr in (('train', items[:1000]), ('test', items[1000:])): os.makedirs(name, exist_ok=True) for i, it in enumerate(arr): blob = json.dumps(it, ensure_ascii=False, sort_keys=True).encode() open(f'{name}/{i:04d}.json', 'wb').write(blob) print(len(items)) EOF I split the data chronologically, not randomly: train holds 1,000 records from 2024-05-07 to 2026-07-15, and test holds the 346 records after 2026-07-16. In production you train a dictionary on the past and compress the future, so the measurement should work the same way. Raw sizes: the training set is 909,086 bytes (909 bytes per record on average), the test set 314,337 bytes (908 bytes on average). The totals differ, but the record profiles match — 909 against 908 bytes — so a per-record comparison is fair. One warning: if you check with du -sh you will see 5.3 MB. That number is filesystem block allocation, not data. A thousand 900-byte files each occupy a 4 KiB block on disk. Every figure in this article is a real byte total read with stat -f '%z'. Step 1: The baseline without a dictionary First, how far do we get with no dictionary at all? Each record is compressed separately — that is the whole point, individually addressable records. mkdir -p out-nodict # --output-dir-flat does NOT create the directory; without this it errors out zstd -19 -q -f test/*.json --output-dir-flat out-nodict find out-nodict -name '*.zst' -exec stat -f '%z' {} + | awk '{t+=$1} END{print t}' 141018 314,337 bytes in, 141,018 bytes out. A ratio of 2.23×. Dropping the level from 19 to 3 gave 143,855 bytes — a 2% difference. That tells you a lot: raising the compression level on a 900-byte file buys almost nothing, because the problem is not how hard the algorithm works but that it has no history to work with. Step 2: Training the dictionary and fooling myself Training is a single command and finishes in the blink of an eye: zstd --train train/*.json -o d-default.dict Trying 5 different sets of parameters WARNING: The maximum dictionary size 112640 is too large compared to the source size 680764! size(source)/size(dictionary) = 6.043714, but it should be >= 10! This may lead to a subpar dictionary! We recommend training on sources at least 10x, and preferably 100x the size of the dictionary! k=1024 d=8 f=20 steps=4 split=75 accel=1 Save dictionary of size 112640 into file d-default.dict Park that warning for now; we come back to it halfway through. zstd told me at training time that my corpus could not carry this dictionary. On first reading I took "subpar" to mean the compression ratio would come out a little lower. Measuring showed it meant something else. Training on 1,000 records took 0.13 seconds. The resulting dictionary is 112,640 bytes, which is zstd's --maxdict default; the CLI manual says so in as many words: "Limit dictionary to specified size (default: 112640 bytes)." The d=8, f=20, steps=4, split=75 lines are the default trainer's parameters — the manual notes that --train is equivalent to --train-fastcover=d=8,steps=4. Now for the appetising part. Let us try the dictionary on the records it was trained on: mkdir -p out-train-dict zstd -19 -q -f -D d-default.dict train/*.json --output-dir-flat out-train-dict Without a dictionary the training set compressed to 401,550 bytes; with the dictionary it dropped to 136,965. A 65.9% gain. If I stopped here and wrote "the zstd dictionary cut output to a third", the number would be correct and the article would be wrong. Step 3: Data it has never seen The same dictionary, applied to the 346 records it never saw in training: mkdir -p out-test-dict zstd -19 -q -f -D d-default.dict test/*.json --output-dir-flat out-test-dict 65602 141,018 → 65,602 bytes. A 53.5% gain, a ratio of 4.79×. Still an excellent result — more than twice as good as the 2.23× without a dictionary. But it sits 12.4 points below the 65.9% the training set reported. That gap is not a fault; it is the definition. The dictionary has absorbed the frequently occurring fragments of the training data, so when it compresses that same data it is looking at its own copy. On a new record, only the patterns that genuinely generalise help: field names, the date format, the URL prefix, frequently used tags. But is the gap really memorisation, or were the two sets simply different in difficulty to begin with? The control is already in hand: without a dictionary, the compression ratio is 2.264× on the training set and 2.229× on the test set — 1.6% apart. The two sets are near-identical work for the compressor. The 12.4-point distance that opens up once the dictionary joins therefore comes from the dictionary's memory, not from a difference in the data. !Diagram The gap follows a rule: the corpus-to-dictionary ratio Seeing the gap at a single dictionary size could have been coincidence. So I repeated the measurement at five sizes — every dictionary trained on the same 1,000 records, then applied to both the training and the test set. | Dictionary | Corpus/dictionary | Gain on training set | On unseen data | Gap | |---|---|---|---|---| | 1,024 B | 888× | 41.0% | 39.1% | 1.9 points | | 4,096 B | 222× | 48.7% | 44.7% | 4.0 points | | 16,384 B | 55× | 55.1% | 48.7% | 6.4 points | | 65,536 B | 14× | 62.1% | 52.1% | 10.1 points | | 112,640 B | 8× | 65.9% | 53.5% | 12.4 points | The gap widens monotonically and predictably as the dictionary grows relative to the corpus. When the dictionary is one-888th of the corpus I fool myself by 1.9 points; at one-eighth, by 12.4. Now back to that warning in the training output. zstd had printed "size(source)/size(dictionary) = 6.043714, but it should be >= 10" on my screen and recommended 100×. The same ratio sits in the library header; inside lib/zdict.h, the note for ZDICT_trainFromBuffer reads: "It's recommended that total size of all samples be about ~x100 times the target size of dictionary." Notice what the warning did not say. It did not say compression would get worse — it said "subpar dictionary". The table shows what that word actually buys you. As the ratio drops I do not compress unseen data worse; quite the opposite, the big dictionary is still the best performer there. What degrades is not the compression but the trustworthiness of the measurement. The ratio governs how badly the number you see on the training set overstates reality. Let us put a figure on it for my corpus. The dictionary was trained on the training set alone: 909,086 bytes. A 100× ratio points at a dictionary of about 9 KB. That lands between the 4 KiB row (gap 4.0 points) and the 16 KiB row (gap 6.4 points) — so obeying the rule caps my self-deception at four to six points, where the default takes it to 12.4. Read that advice not as a growth recipe but as an honesty margin: when you hit the ratio, the number you see on the training set is close to the truth. The further below it you go, the more you owe your own measurement. Step 4: Where to stop growing the dictionary So does a bigger dictionary not compress better after all? On unseen data it does, but the returns shrink fast. Look at the fourth column on its own: - The first 1 KiB buys 39.1 points over the no-dictionary baseline. - The remaining 109 KiB adds only 14.3 points more. In other words, the dictionary's first kilobyte delivers roughly 73% of the total benefit. Choosing 16 KiB over 110 KiB costs just 4.7 points on unseen data. I wanted to continue with "so that is a factor of seven in memory", because 112,640 divided by 16,384 is seven. Then the numbers came in, and my own thesis caught me. Peak RSS from /usr/bin/time -l while compressing a single record: 2,016 KiB with no dictionary, 1,616 KiB with the 16 KiB dictionary, and 1,616 KiB with the 110 KiB one. No difference between the two dictionary sizes, and both below the no-dictionary run. At level 19 the match-table allocations dwarf the dictionary's raw bytes; at CLI scale, dictionary size simply does not show up in RSS. The lesson: I had invented the memory justification for shrinking the dictionary. The real justification is not memory but carriage — the dictionary is a file that has to reach every reader, be backed up, and be versioned. Working through the library API the table may look different; measure it in your own stack then. I also tested the amount of training data as a separate variable, pinning the dictionary size at 16 KiB and varying the record count: | Training records | Output on unseen data | |---|---| | 25 | 78,671 B | | 100 | 73,971 B | | 300 | 75,437 B | | 1,000 | 72,293 B | Two things stand out. First, even the dictionary trained on 25 records captures 91% of the gain that 1,000 records produce — and that dictionary is only 11,081 bytes, because 25 samples cannot fill 16 KiB. Second, the curve is not monotonic: going from 100 records to 300 made the result worse (73,971 → 75,437). I will not present that as a law — it is one measurement on one corpus — but it does show there is no guarantee behind the intuition that "more samples always mean a better dictionary". When you grow the training set, measure the gain instead of assuming it. Step 5: The alternative nobody measures — batching This is the point to step back. A dictionary solves the "no history" problem. But that same problem has a far cheaper solution: compress records in batches rather than one at a time. Then the history comes from the earlier records in the batch instead of from a dictionary. Same 346 records, same level, the only variable being batch size: | Batch | Without dictionary | With dictionary | |---|---|---| | 1 record | 141,018 B | 65,602 B | | 10 records | 71,962 B | 52,754 B | | 50 records | 56,988 B | 48,886 B | | 100 records | 53,560 B | 47,742 B | | Single stream | 49,339 B | 46,882 B | Read that table once more. Batches of 50 with no dictionary (56,988 B) beat per-record compression with a 110 KiB trained dictionary (65,602 B). No dictionary maintenance, no versioning, no distribution — just grouping the records. This does not make dictionaries worthless; it clarifies what they are for. A dictionary is the answer when you cannot batch. If you need per-record random access — reading a row from a KV store, pulling a single message off a queue — there is no such thing as a batch of 50, because reading one record would mean decompressing fifty. That is exactly when a dictionary turns 2.23× into 4.79× while keeping every record independently decodable. The two do combine, and the dictionary keeps paying even in batched layouts: on the single stream it takes 49,339 down to 46,882, another 5%. But note what that 5% costs you — dictionary distribution, versioning and DictID tracking. Adding a dictionary when you could batch instead trades a marginal gain for operational weight; when per-record access is mandatory, the dictionary is the lever that turns 2.23× into 4.79×. Step 6: What happens if you lose the dictionary Now the part that will hurt you in production. Data compressed with a dictionary cannot be decompressed without it. This is not a performance question, it is an availability question. zstd -l -v single-record.zst DictID: 1312853601 Window Size: 939 B (939 B) Compressed Size: 148 B (148 B) Decompressed Size: 939 B (939 B) Check: XXH64 55ceeb5a The frame header carries a DictID. RFC 8878 defines the field as "the ID of the dictionary required to properly decode the frame", and the zstd manual says a 4-byte random number ID is created by default. Try to decompress without it, or with the wrong dictionary: single-record.zst : Decoding error (36) : Dictionary mismatch Both attempts exit with status 1 and write no output file. That is good news: zstd errors out rather than silently producing wrong data. Using a raw file as a dictionary -D does not insist on a trained dictionary. Per RFC 8878, zstd also accepts "raw content" dictionaries — free of any format restriction beyond a minimum of 8 bytes. So you can concatenate a few sample records and hand that over as a dictionary. I tried it with five separate consecutive bundles of five records each from the training set (0–4, 100–104, 200–204, 300–304, 400–404; all between 4.2 and 4.8 KB). Their outputs on the test set spread from 92,080 to 95,614 bytes. The trained 4 KiB dictionary gives 77,948 — 15% smaller than even the best raw bundle. I am not quoting a single ratio from one bundle, because which records you happen to pick moves the result by 3,500 bytes; what training really adds is that it removes that lottery. The raw dictionary's second difference matters more: DictID: 0 No identity. When I tried to decompress with the wrong raw dictionary I still got an error — but a different one: Decoding error (36) : Restored data doesn't match checksum What caught it was not the DictID but the content checksum. RFC 8878 defines it as an XXH64 digest of the decoded data, and the zstd manual notes the check is enabled by default. So what happens if you turn it off? zstd -19 --no-check -D rawA.dict record.json -o nc.zst zstd -d -D rawB.dict nc.zst -o nc.out # wrong dictionary echo $? 0 Exit status zero. No error. And nc.out was written — 939 bytes, the same size as the original. Here is what is inside it: {ified": "2026-04-06T00:00:00.000Z", "date_published": "2026-04-06T00:00:00.0 gelistiriciler-icin-dijital-guvitle"-5-kolayle"-maliyet-dengesie_modified": ... It is not valid JSON, but it looks like JSON: real dates, real slug fragments, familiar field names. In a logging pipeline or a bulk restore, this has a very good chance of slipping through unnoticed. You will not get this outcome with every wrong dictionary: the offsets the decoder reads out of the dictionary have to land in bounds, and with a sufficiently smaller wrong dictionary you get "Corrupted block" instead. My two raw dictionaries were close in size (4,201 and 4,427 bytes), which opened the silent-corruption path. Three conditions must hold at once: a raw dictionary (DictID 0), plus --no-check, plus the wrong dictionary. Each one can be chosen for a perfectly reasonable-sounding motive — "we know which dictionary it is", "the checksum burns CPU", "we refreshed the dictionary file during deploy". With a trained dictionary the DictID stops you at the first hurdle; with a raw dictionary your only defence is the checksum, so leave it on. Do dictionaries go stale? One question remains: does a dictionary trained on old data decay over time? On the first attempt I trained two dictionaries — the oldest 300 records and the newest 300 — both 16 KiB, both applied to the same 47 records from October 2026 (42,162 bytes). The stale one gave 11,485 bytes and the fresh one 10,842, and I was about to write "a two-and-a-half-year-old dictionary still delivers 93% of the fresh one's benefit". Something nagged before I wrote it: how much does which 300 I picked move the answer? So I split the pool in half and trained four dictionaries, taking 300 records from the start and the end of each half. | Dictionary | Training range | Output | |---|---|---| | none | — | 19,619 B | | start of the old half | 2024-05 → 2026-04 | 11,485 B | | end of the old half | 2026-04 → 2026-05 | 11,366 B | | start of the new half | 2026-05 → 2026-07 | 11,423 B | | end of the new half | 2026-07 → 2026-09 | 10,842 B | The first three rows sit within 119 bytes of one another. Two years separate them and the difference is almost nothing. Only the last row breaks away: the three months immediately before the target do 524 bytes better. So "does the dictionary go stale" was the wrong question. Choosing between 2024 data and mid-2026 data moves the target by 62 bytes — noise. The only measurable effect is adjacency: the period right next to the target does slightly better than any distant one. My first attempt's "93%" was arithmetically right but overstated the effect, because I had picked the two extremes. For my data this makes sense: what the dictionary captures is largely schema — field names, date format, URL prefix. As long as the schema holds, calendar age carries no weight. If your schema or vocabulary genuinely shifts, the result will differ; measure rather than assume — and when you measure, do not repeat my first-round mistake, try several slices instead of one. The practical consequence: retraining the dictionary on a cron job creates more problems than it solves, because every new dictionary means a new DictID, and you must keep the old ones around to read the old data. What about speed? I repeated each measurement 10 times and took the minimum and median (Apple M4 Pro, macOS 26.6.2, zstd 1.5.7; 346 records in a single zstd invocation): | Measurement | Minimum | Median | |---|---|---| | Compress -19, no dictionary | 111 ms | 114 ms | | Compress -19, with dictionary | 127 ms | 138 ms | | Compress -3, with dictionary | 65 ms | 71 ms | | Decompress, no dictionary | 57 ms | 67 ms | | Decompress, with dictionary | 58 ms | 68 ms | The dictionary slows compression down by 14% at the minimum. On decompression the 1 ms difference between the two rows is smaller than the run-to-run variation within a single measurement — so the dictionary's decompression cost is unmeasurable on this data. Do not carry that row straight into production. The measurement decompresses 346 records in a single zstd invocation: the dictionary is read and digested once, then amortised across 346 records. The scenario this article recommends dictionaries for is the opposite — per-record random access, one record per read. On the library side, a naive implementation that builds a fresh ZSTD_DDict from 112,640 bytes on every read does far more work than decoding the 939-byte record itself. The right shape is to build it once with ZSTD_createDDict and reuse it through ZSTD_DCtx_refDDict; the dictionary object is read-only and shareable across threads. My CLI figure is what that correct setup looks like, and it says nothing about code that rebuilds the dictionary on every read. Picking the level is the more effective lever: -3 with the dictionary gives 73,202 bytes in 65 ms, while -19 gives 65,602 bytes in 127 ms. Twelve per cent more output for half the time. On a write-heavy path, -3 is almost always the right call. A decision framework !Diagram As a checklist: 1. Try batching first. Batches of 50 without a dictionary beat a 110 KiB dictionary. If you do not need random access, the job ends here. 2. Size the dictionary against the corpus. Around 1% of the training corpus is a good starting point for --maxdict; that is zstd's own advice, and it prints a warning when you ignore it. 3. Measure on unseen data. The number you get on the training set can mislead your capacity plan by up to 12 points. 4. Start small. Even a dictionary trained on 25 records delivers 91% of the benefit; do not assume the returns of a large training set, measure them. 5. Store the dictionary as durably as the data. Losing the dictionary means losing the data. Record the DictID; include the dictionary in your backup plan. 6. Stay away from --no-check, especially with a raw dictionary. The CPU you save is not worth silently corrupted data. 7. Justify retraining. With a stable schema, calendar age cost me almost nothing; every new dictionary is a new DictID you have to carry. Three topics this article does not cover will meet you on the way to production; at least know their names. First, dictionary distribution and version coexistence: DictID catches the wrong dictionary but it does not deliver the right one — where you put it (sidecar object, id-to-blob table, baked into the binary) and how long you retain old versions is your design. Second, the library API: this article measures the CLI, while production uses ZSTD_createCDict/createDDict, and the cost model changes there. Third, and least discussed: a shared dictionary is a side channel. In a multi-tenant store where attacker-influenced data and secrets share one dictionary, the compression ratio leaks information — the same logic as CRIME and BREACH. Use a dictionary per tenant, or keep secrets out of the dictionary path. If you are thinking about storage cost in layers, the piece where I worked through the same logic on the logging side — Tiered Log Retention with Grafana Loki — is a good companion: there too, the gain came not from a single setting but from what question you were asking of the data. Closing thought The real lesson from this experiment is not about compression. The zstd dictionary genuinely does a good job — going from 2.23× to 4.79× is nothing to sniff at, and it does so while preserving per-record random access. The lesson is this: if you measure the benefit of an optimisation on the very data you derived it from, what you are measuring is not benefit but memorisation. On my corpus the difference was 12.4 points, and the gap closed steadily as I shrank the dictionary relative to the corpus. zstd had even printed it on my screen at training time; I read "subpar dictionary" as a remark about the compression ratio and moved on. What the warning was really describing was how misleading my measurement would be. Build a capacity plan on the wrong side of that gap and you will spend six months from now wondering why the disk is 30% fuller than you expected. Separating the two costs nothing: split the data in half, train on one half, measure on the other. Official Sources - zstd CLI manual (zstd.1.md) — --train, the --maxdict default (112640 bytes), -D, --[no-]check and --dictID behaviour. - lib/zdict.h — the zstd dictionary builder header — sample count guidance for ZDICT_trainFromBuffer and the "~x100 times the target size of dictionary" recommendation. - RFC 8878 — Zstandard Compression and the application/zstd Media Type — the Dictionary_ID frame header field, the "raw content" dictionary definition, and the XXH64-based Content_Checksum.

Scaling Tick & Order-Book Data Pipelines for Trading Analytics - Data collection: resilient gateways and canonical normalization - Designing storage for time-series and order-book snapshots - Compression, partitioning, and retention that minimize cost - Querying at scale: indexing, aggregation, and benchmark recipes - Practical checklist for deploying a production pipeline Tick-level market data outgrows naive storage fast: message bursts, trade corrections, and microsecond-level timestamps turn ad-hoc pipelines into operational liabilities. The right architecture treats the market feed as the single source of truth, separates event storage from snapshot storage, and designs tiering and compression before terabytes arrive. You’re seeing the symptoms every quant/dev team recognizes: dashboards that slow to a crawl on market-open days, backtests that disagree with live fills because of replay errors, and SRE tickets for recovery after a missed sequence number. Those problems all trace back to the same root causes: unpredictable ingest, ambiguous canonical schema, and a single-tier storage model that can’t trade off cost versus access. The rest of this piece describes practical, field-tested patterns for building a scalable tick data pipeline and order book storage layer using modern time-series DBs, columnar archives, and retention tiering. Data collection: resilient gateways and canonical normalization Why it matters - Gateways and feed handlers are the firewall between noisy exchange formats and your analytics stack. Treat them as stateful, deterministic components that enforce integrity, not as simple parsers. Core patterns - Owned canonical model. Convert every incoming vendor/exchange format to a small, strict canonical event model. Minimal required fields for ticks and book events: symbol, msg_type (trade|quote|book_update|snapshot|cancel|delete), price, size, side, order_id (if present), seq (exchange sequence), exchange_ts (exchange-provided), recv_ts (local), and raw (opaque original). Keep the canonical model intentionally compact and typed; use enums for msg_type and side. - Deterministic gateway topology. Put feedhandlers closest to the network (ideally on hosts with PTP-synced NICs), parse binary protocols (SBE/FAST/ITCH/OUCH), validate sequence numbers, enrich with recv_ts, and publish canonical messages to a durable streaming buffer (Kafka/Kinesis). The FIX community resources and SBE/FAST standards are the right place to start when you design feed handlers. (fixtrading.org) - Hardware timestamps and PTP. For microsecond/nanosecond fidelity, use NICs and switches that support hardware timestamping and deploy PTP (IEEE 1588) for synchronizing clocks across capture hosts. Relying on OS timestamps alone creates nondeterministic order and complicates reconstruction. (ntp.org) - Buffer + replay layer. Always put a durable, replayable buffer between parsing and storage. Kafka provides idempotent producers and transaction semantics that let you guarantee write semantics across restarts; enable enable.idempotence=true and acks=all for production feed pipelines. (confluent.io) Edge cases you must design for - Out-of-order messages: implement a bounded reorder buffer keyed by (symbol, source) that reorders by seq or exchange_ts before committing. Make the window configurable per-feed. - Missing sequence numbers: mark holes and request snapshots from the exchange or vendor; persist hole metadata so you can later reconcile gaps during EOD processing. - Duplicates: dedupe on (source, symbol, seq) or a hash of (raw_message); make dedupe idempotent and cheap (Bloom filters + short-lived lookups). - Corrections/reprints: record corrections as separate events (with a corr_origin field pointing to original seq) rather than mutating historical rows; that preserves auditability. Implementation sketch (Python -> Kafka) # python pseudocode: parse -> canonical -> kafka from confluent_kafka import Producer import json, socket, struct, time p = Producer({ "bootstrap.servers":"kafka:9092", "enable.idempotence": True, "acks":"all", "linger.ms": 5 }) def on_feed_packet(buf, src): msg = parse_native_protocol(buf) # SBE/FAST/ITCH parser in C++/Rust canonical = { "symbol": msg.symbol, "msg_type": msg.type, "price": msg.price, "size": msg.size, "side": msg.side, "order_id": msg.order_id, "seq": msg.seq, "exchange_ts": msg.ts, "recv_ts": time.time_ns() } p.produce("canonical-feed", key=canonical["symbol"], value=json.dumps(canonical)) p.poll(0) > Important: set the feedhandler language to a compiled runtime (C/C++/Rust) for binary parsing and NIC-level packet capture; keep Python/Ruby for orchestration and downstream analytics. Designing storage for time-series and order-book snapshots Two complementary storage models - Event model (append-only message log). Store raw, canonical feed messages as the immutable source-of-truth. This is compact, cheap to append to, and ideal for full reconstructions and compliance replays. - Snapshot model (materialized view of ladder). Store periodic snapshots or top-N level snapshots for fast queries (TCA, markouts, front-running detection). Snapshots are larger but accelerate common analytical workloads (ASOF joins, VWAP markouts). Schema examples (TimescaleDB / SQL) -- event model (hypertable) CREATE TABLE orderbook_events ( time TIMESTAMPTZ NOT NULL, symbol TEXT NOT NULL, msg_type TEXT NOT NULL, order_id BIGINT, side CHAR(1), price DOUBLE PRECISION, size BIGINT, seq BIGINT, exchange_ts TIMESTAMPTZ, recv_ts TIMESTAMPTZ DEFAULT now(), raw JSONB ); SELECT create_hypertable('orderbook_events','time', chunk_time_interval => INTERVAL '1 day'); -- snapshot model for top-N (arrays for levels) CREATE TABLE orderbook_snapshots ( time TIMESTAMPTZ NOT NULL, symbol TEXT NOT NULL, bid_prices DOUBLE PRECISION[], bid_sizes BIGINT[], ask_prices DOUBLE PRECISION[], ask_sizes BIGINT[], depth INT ); SELECT create_hypertable('orderbook_snapshots','time', chunk_time_interval => INTERVAL '1 day'); Schema notes and tradeoffs - Arrays vs normalized levels: use arrays for fast read of full-ladder where you read every level together; use row-per-level when analysts frequently filter by price level. For many production analytics (ASOF join TCA), top-5/top-10 arrays are efficient. - Hybrid strategy (recommended): store every incremental orderbook_event as the canonical log, and also persist periodic orderbook_snapshot rows (e.g., 1s for active tickers, 1m for thin names). Snapshots speed up ASOF joins and reduce replay costs. - Example datasets such as LOBSTER present the same pairing of message and orderbook files — you can mirror that structure: an append-only messages stream and a separate snapshot product for quick access. (lobsterdata.com) kdb+ operational pattern - Use the classical tickerplant → RDB → HDB architecture: the tickerplant logs messages, the RDB serves the current day in memory, and the HDB is the historical store on disk. kdb+’s tick pattern remains the de-facto approach for ultra-low-latency tick analytics. (code.kx.com) Compression, partitioning, and retention that minimize cost Partitioning & chunk sizing - Partition primarily by time. Make time your first-class partition key and choose a chunk interval that fits your memory/IO profile. Timescale’s guidance: set chunk_interval so a chunk is roughly 25% of main memory (e.g., if you write ~10 GB/day and have 64 GB RAM, prefer 1-day chunks). That reduces frequent disk reads during recent-data queries and keeps chunk creation overhead manageable. (docs.timescale.com) - Secondary partitioning: when query patterns filter heavily by symbol, enable chunk skipping range stats on the symbol or other correlated columns (enable_chunk_skipping) to allow the planner to prune irrelevant chunks quickly. Storage tiers and retention design (typical) - Hot tier (0–7 days): recent tick-level data in a low-latency store (in-memory DB or fast SSD-backed TSDB like kdb+/RDB, QuestDB, or Timescale with uncompressed hypertables). - Warm tier (7–90 days): compressed columnar store (Timescale columnstore or Parquet files on fast object store), ready for ad-hoc analytics. - Cold tier (90 days+): compressed Parquet (ZSTD) on object storage / Glacier for compliance and occasional audits. Compression choices and trade-offs - Columnar + Parquet for historical blobs. Use Parquet with ZSTD (or LZ4_RAW for fastest decompression) to balance storage and query time; Parquet explicitly supports ZSTD, LZ4_RAW, GZIP, SNAPPY and documents trade-offs between codecs. (parquet.apache.org) - Zstandard is a modern general-purpose algorithm with an excellent speed/ratio tradeoff; use lower zstd levels for hot, higher levels for archival. (facebook/zstd) - For in-DB columnar compression (Timescale’s hypercore/columnstore), rely on delta/delta-of-delta for timestamps and XOR-style float compression (derived from Gorilla), which gives high ratios for ordered time series. That’s how Timescale achieves strong compression on numerical time-series columns. (docs.timescale.com) File size & partition granularity - Avoid many tiny files. Aim for Parquet files in the 128MB–512MB range to keep object-store queries efficient; perform regular compaction jobs to merge small files produced by streaming ingestion into efficient read-optimized files. Cloud/EMR best practices call this out as a major performance lever. (aws.github.io) Retention & lifecycle automation - Move data across storage classes via lifecycle policies (S3 lifecycle rules or equivalent). Use S3 Intelligent-Tiering or explicit transitions to Glacier/Deep Archive for long-lived archives, and be mindful of minimum storage duration and restore times when choosing class transitions. (aws.amazon.com) (docs.aws.amazon.com) Small worked example (cost-aware retention) - Keep raw events for the last 30 days in your TSDB (hot+warm), convert older daily chunks to Parquet and move to S3 Standard-IA after 30 days, then to Glacier Deep Archive after 1 year. Make restore paths explicit for compliance requests and automate compaction and partition repair as part of your nightly ETL. Querying at scale: indexing, aggregation, and benchmark recipes Indexing & query shaping - Time-first indexes. Your planner must see time first; then place symbol second (composite index (symbol, time DESC)) for most backtests and TCA queries. - Chunk skipping / min-max statistics. Enable chunk/min-max range stats on correlated columns that appear frequently in WHERE clauses (Timescale’s enable_chunk_skipping) so the engine prunes chunks quickly during scans. (docs.timescale.com) - Materialized roll-ups. Precompute continuous aggregates for common windows (1s/1m/1h) and combine them with recent raw data for "real-time aggregation" queries. Use continuous aggregates (Timescale) or materialized views (kdb+/derived tables) to avoid repeated full scans. (docs.timescale.com) Analytics patterns - ASOF joins (nearest prior match). ASOF/join semantics are essential to pair trades with the latest order-book snapshot. Some TSDBs (QuestDB, kdb+) provide built-in ASOF semantics; otherwise implement efficient rolling-window joins that index by symbol and time. QuestDB documents efficient ASOF join usage for TCA workloads. (questdb.com) - Pre-aggregations for TCA: maintain materialized results for VWAP windows, execution slippage, and markouts to reduce read-time pressure. Benchmark recipes (what to measure) - Ingest throughput (rows/sec sustained, peak burst handling). - Query latency P50/P95/P99 for representative queries: symbol-range scan, day-of-symbol ASOF join, 1-day aggregates. - Storage efficiency (raw bytes -> compressed bytes) per table and per retention tier. - Recovery time for replaying missing sequences (minutes to rehydrate recent HDB segment). Benchmarks and what vendors claim - kdb+ is architected around the tick pattern (tickerplant → RDB → HDB) and remains widely used where sub-ms analytics are required; it is a natural fit for the classic tick storage and replay architecture. (code.kx.com) - Alternative high-performance TSDBs (QuestDB) advertise high ingestion rates and native Parquet export for archival workflows; their ASOF join features can simplify trade-to-book pairing at scale. Use vendor claims as a starting point and run your workload-specific benchmarks before selecting a primary store. (questdb.com) Quick comparison table (high-level) | Concern | Event log (append-only) | Snapshot (periodic) | |---|---:|---| | Write cost | Low | Higher | | Replay cost to reconstruct book | Needs replay | Immediate | | Query latency for ASOF join | Higher | Lower | | Best for | Compliance, full reconstruction | TCA, fast analytics | Practical checklist for deploying a production pipeline Operational checklist (ordered) 1. Feed & time integrity - Deploy PTP-synced NICs and timestamp capture on feedhosts. (ntp.org) - Implement per-feed sequence validation and hole tracking in the gateway. 2. Canonical model & contract - Define a compact canonical event schema and enforce it at feedhandler output. - Commit schema to a registry (JSON Schema / Avro / Protobuf) and enforce compatibility. 3. Buffer & durability - Publish canonical events to Kafka with enable.idempotence=true, acks=all. Test exactly-once paths for your processing pipeline. (confluent.io) 4. Storage & tiering - Implement hypertable + chunk policy (or kdb+ tick) for hot data; convert chunks to columnar store after N days. Tune chunk interval to keep one chunk ≈ 25% RAM. (docs.timescale.com) 5. Compress & archive - Export historical chunks to Parquet with ZSTD compression for cold storage; target 128–512MB files and run compaction jobs nightly. (parquet.apache.org) (aws.github.io) 6. Index & aggregate - Create composite indexes on (symbol, time) and enable chunk skipping on high-cardinality secondary columns. - Materialize continuous aggregates for the queries your traders run every day. (docs.timescale.com) 7. Monitoring & SLOs - Monitor ingest latency, reorder buffer sizes, and chunk-creation rates. - Define SLOs: ingest durability (99.99%), replay time for last 24h (minutes), bulk export latency (hours). 8. Recovery & reconciliation - Automate hole reconciliation: compare logged exchange sequence ranges, fetch snapshots for missing periods, and run a deterministic replay to fill gaps. 9. Compliance & audit trail - Keep raw canonical raw payloads for the minimum compliance period; store audit metadata describing any corrective patches (reprints/cancels). 10. Benchmark & runbooks - Maintain reproducible benchmark harnesses (ingest generator + replay) and run them monthly; keep an operational runbook for EOD, failover, and restore procedures. > Important: Keep the append-only canonical log as the immutable source-of-truth; all snapshots and roll-ups must be derived artifacts with traceability back to the canonical log. Last thought: build your pipeline so you can re-create the truth from first principles—append-only canonical events, strict timestamps, and durable, compressed archives—then optimize for read patterns with snapshots, continuous aggregates, and storage tiering. The moment your pipeline can answer "what happened exactly at 09:30:00.123456789 UTC for symbol X" without ambiguity, you have built infrastructure that supports both trading analytics and regulatory audits. Sources: Realtime database – Starting kdb+ (kdb+ tick architecture) - Describes the kdb+ tickerplant / RDB / HDB architecture used for tick ingestion and real-time queries. (code.kx.com) Improve hypertable and query performance (TimescaleDB) - Guidance on choosing chunk_interval, chunk-sizing heuristics (e.g., 25% memory rule) and partitioning strategy. (docs.timescale.com) Parquet file-format compression documentation - Supported codecs and recommendations for Parquet compression (ZSTD, LZ4_RAW, Snappy, GZIP). (parquet.apache.org) Zstandard (zstd) GitHub repository - Zstandard reference implementation, performance characteristics and tuning options for real-time compression. (facebook/zstd) Amazon S3 – Object storage classes (Overview) - Storage-class options (Standard-IA, Intelligent-Tiering, Glacier) for tiering archived tick data. (aws.amazon.com) FIX Trading Community – Standards and SBE/FAST references - Official FIX standards, SBE/FAST encoding guidance and recommended practices for market messages. (fixtrading.org) NTP.org reference: PTP (IEEE 1588) vs NTP discussion and timestamp capture principles - Technical overview of PTP vs NTP, hardware timestamping and why PTP is used for sub-microsecond time sync in trading systems. (ntp.org) Exactly-once semantics in Apache Kafka (Confluent blog) - Explanation of idempotent producers, transactions and exactly-once processing guarantees for Kafka-based pipelines. (confluent.io) LOBSTER dataset – output structure and example message/snapshot pairing - Academic-level example of separate message (events) and orderbook (snapshot) outputs used in microstructure research. (lobsterdata.com) QuestDB for market data & ASOF join examples - Vendor documentation showing ASOF join usage and high-ingest design for market data workloads. (questdb.com) AWS EMR/Big Data best practices – avoid small files and compact Parquet - Practical guidance on file size targets and compaction to avoid S3/listing overheads. (aws.github.io) TimescaleDB – About compression methods (hypercore / columnstore) - Details on delta/delta-of-delta, XOR-based float compression, and Timescale’s columnstore behaviors for time-series compression. (docs.timescale.com) Transitioning objects using Amazon S3 lifecycle (details) - Lifecycle rule behavior, minimum retention durations, and practical considerations when transitioning objects to Glacier/Deep Archive. (docs.aws.amazon.com)

Release Playden for PS5 0.1.0 · leolobato/playden-ps5 First public release of Playden for PS5: play your Steam library's Windows games on a jailbroken PS5. Playden for PS5 Play your Steam library's Windows games on a jailbroken PS5. Playden is a launcher that runs on the PS5, adapted from Playden for macOS. You sign in to Steam with a QR code, browse your library, download games, and play them. The Windows games run through prospero-win, a port of Wine to the PS5. Each game gets its own private Windows environment and its own tile on the PS5 Home screen. > [!WARNING] > Alpha software. Playden is an early, experimental project. It signs in to your real Steam account, downloads > games with your licenses, and reads and writes your Steam Cloud saves. Read [Before you use it](#before-you-use-it). > If you are not comfortable with these risks, do not use it.  Features - Steam sign-in with a QR code. Approve the sign-in in the Steam mobile app. No password typed on the console. - Your whole Steam library. Search, sort, favorites, collections, ratings and notes, with cover art and store details. The library works offline from a cache. - Downloads from Steam. Queue games and watch progress. - Each game is its own PS5 app. It gets a tile on the Home screen, so you can start it without Playden. - A private Windows environment per game ("bottle"), so games do not share settings or files. - Steam Cloud saves. Playden checks your Cloud saves before a game starts and syncs them after it ends. If the console and the Cloud disagree, you choose which one to keep. You can turn the automatic sync off in Settings and sync by hand from the Saves tab. - Per-game settings. Graphics, frame rate, resolution, controller mapping, and how the game starts: pick any program from the game folder, edit its arguments and library overrides, or try a game Playden flagged anyway. A game's settings can be exported as a recipe ([format](docs/community-recipes.md)). - Graphics. Direct3D 8–11 through DXVK, OpenGL through Mesa's Zink, and Vulkan, all on the PS5 GPU's Vulkan driver. - Gamepad support. XInput, DirectInput, Raw Input, Windows.Gaming.Input and the Windows multimedia joystick API, with rumble. Games see the DualSense as an Xbox controller. - Mouse and keyboard. Plug a USB mouse and keyboard into the PS5, or use a controller preset: the left stick as a mouse, or the D-pad as the arrow keys. - Install to internal or external disks. Console storage, or an exFAT USB drive formatted on a Mac or PC. See the [compatibility list](COMPATIBILITY.md) for the games we have tested. Planned - Steam game updates - Other stores (GOG, Epic) - Support for game launches (Rockstar, EA, Ubisoft) - Community recipes in the preset chooser Before you use it Playden is alpha software. It is not affiliated with, endorsed by, or supported by Valve. Know these points before you sign in: - It is an unofficial Steam client. Playden talks to Steam's servers directly, the way the Steam client does. Valve does not support third-party clients. We know of no account problems from this, but the risk is yours to accept. - Your Steam session stays on the console. Playden stores a Steam sign-in token on the PS5, without encryption. On a jailbroken console, anyone with file access (for example through FTP) can read it. Signing out in Playden deletes the token from the console. - It writes to your Steam Cloud saves. Playden uploads saves after you play. A bug can overwrite good saves with older or broken ones. Back up saves you care about. When Playden shows a conflict, read it before you choose. - It needs a jailbroken console with root-level helpers. A running game can get elevated rights on the console. Run only games you trust. - Things will break. Expect games that fail to start, crashes, and the occasional console restart. Keep your console's own data backed up. Requirements - A jailbroken PS5 (for example with the WebKit autoloader). We test on firmware 13.60. Playden does not block other firmware versions, but they are untested. Please give your firmware in a test report. - These payloads loaded after each boot: - kstuff — required. Without it, every game fails with error CE-100008-9. - ShadowMountPlus 1.7-beta5fix1 — puts each game's tile on the Home screen. 1.7beta2 also works. - elfldr, the ELF loader on port 9021 — runs the root helper that gives a game access to its files. Most jailbreak loaders start it for you. - nanoDNS, if your console's DNS is set to 127.0.0.1 (as many jailbreak guides suggest) — without a local DNS server, Playden can't reach Steam. - For installing and updating Playden: an FTP server such as zftpd (port 2120), and optionally ps5-web-file-manager (port 8888). - A Steam account with Windows games. - A computer with Python 3 on the same network as the PS5, to copy Playden and the kit to it. - To build it yourself (optional): a Mac with Apple Silicon. Linux should also work, but we have not tested it yet. See [Build from source](#build-from-source). Tested with kstuff-lite 1.11, ShadowMountPlus v1.7-beta5fix1 and v1.7beta2, elfldr v0.26, zftpd 1.6.0 and ps5-web-file-manager v1.9. Other versions can work, but these are the ones we know. How it works - Playden does not run games itself. It downloads a game, prepares it, and hands it to a separate PS5 app made for that game. That app starts Wine with the game's settings. Because each game is its own app, it has its own tile on the PS5 Home screen, and it is isolated: its files, settings and crashes stay inside it. - The runtime kit is separate from Playden. The kit holds Wine for PS5, the graphics drivers, a clean Windows environment and the per-game app shell. Playden checks that the installed kit is complete before it uses it. Install Download both files from the latest [release](../../releases): - playden-ps5-0.1.0.zip: the Playden app (FAKE77031) and the console tools used below. - playden-kit-h-r6.zip: the runtime kit (hl2-fw1360-dev1), with its source list and license texts. You need Python 3 on your computer for the tools. Unzip both files, then run the commands from the folder that holds tools/. Install Playden 1. Copy the app to the console over FTP. --install creates /data/homebrew/FAKE77031 and its folders, so it also works on a console that has never had Playden (or was just reinstalled): python3 tools/console_title_deploy.py --host <PS5 IP> deploy --install FAKE77031 --backup backups/<name> Copying the FAKE77031 folder by hand with an FTP client or the web file manager also works, but then make /data/homebrew, the FAKE77031 folder and its logs and playden-data folders writable by everyone (chmod 777). Playden writes there as the app's own user, and a folder created over FTP is usually 755: Playden then stays on its splash screen and writes no logs, can't install games ("cannot create install folder"), or can't finish preparing one ("package staging and destination must support one atomic rename"). 2. Wait about 10 seconds. ShadowMountPlus adds the Playden tile to the Home screen. 3. Install the runtime kit (next section). 4. Start Playden and sign in to Steam. To update later, run the same command without --install. It copies only the changed files, keeps a backup of what it replaced, and puts those three folders' modes right if they changed. Install the runtime kit This release uses kit H r6. Upload the kit folder from playden-kit-h-r6.zip: python3 tools/console_kit_upload.py hl2-fw1360-dev1 --host <PS5 IP> --replace The upload checks every file before it switches to the new kit, and creates /data/playden/kits on a console that doesn't have it yet. Playden accepts only the kit version it was tested with. Games pick up a new kit when you press Play. The kit is built on a Linux x86-64 machine or VM from prospero-win, Wine, Mesa and DXVK. To build it yourself, see [Build the kit](CONTRIBUTING.md#build-the-kit). The scripts are a record of how we built it, not a turnkey build yet, so most people should use the release download. Build from source What you need - macOS on Apple Silicon. Linux should also work, but we have not tested it yet. - Xcode command line tools. - Homebrew packages: brew install cmake ninja pkgconf sdl2-compat llvm@18 python - About 2 GB of free disk space. - A checkout path without symbolic links (for example, not under /tmp). Get the source git clone --recursive https://github.com/leolobato/playden-ps5.git cd playden-ps5 scripts/fetch-deps.sh fetch-deps.sh downloads the prebuilt PS5 SDKs into deps/ and checks each download against the SHA-256 in dependencies.lock.json. Build the PS5 package scripts/build-lapy-helper.sh scripts/build-homebrew-ps5.sh The result is the Playden app, build/homebrew-ps5/dist/FAKE77031. Install it as in [Install Playden](#install-playden), with that folder in place of FAKE77031: python3 tools/console_title_deploy.py --host <PS5 IP> deploy --install build/homebrew-ps5/dist/FAKE77031 \ --backup build/console-backups/<name> Try it on your Mac first (optional) Playden also builds as a desktop app, which is useful to look at the interface without a console: scripts/build-homebrew.sh cmake --build --preset homebrew-host build/homebrew-host/playden-homebrew --simulation --scenario signed-out Known issues - New games can stay at Waiting for the PS5 after you quit Playden when ShadowMountPlus stops registering titles. Reload ShadowMountPlus. Until the game is registered, its Cloud saves are not synced before its first launch. - If every game hangs on its loading spinner, the ELF loader has stopped. Reload it. - Some controller games show a mouse pointer in a corner of the screen. - Playden can crash if you open a game page in the first seconds after start-up. - 32-bit games can run through an x86 translator, which is slower than native. For 32-bit OpenGL games this includes Mesa itself, so heavy ones can be slow. - After you update Playden, it can ask you to sign in to Steam again. - Steam can turn down a saved sign-in that worked before. Playden tries again the next time it starts; if it keeps happening, use Settings › Account › Sign in again: it keeps your library, favorites and settings. - Godot games (Slay the Spire 2) need --rendering-driver opengl3 in Game settings › Extra arguments. Playden picks Godot's Vulkan renderer, which did not work on this kit. - Two game settings, "120 Hz output" and "OpenGL worker thread", are not available with this kit. - When you close a game from the PS5 menu, it can stay suspended. Pressing Play again resumes it instead of starting it again. See the [compatibility list](COMPATIBILITY.md) for what we know about each game. Testing games and reporting problems Every test helps, including games that work. To report one: 1. If the game did not work, start Playden once more, so it copies the game's log. 2. Collect the logs from your computer: python3 tools/collect_logs.py --host <PS5 IP> It lists the games that have a log, the most recently run first, and asks which one to collect. The script reads only Playden's logs, never your Steam sign-in. It hides your Steam account name and SteamIDs, and writes a zip with a summary of your firmware and kit versions. 3. [Open a game test report](../../issues/new?template=test-report.yml) and attach the zip. See a [sample report](docs/sample-test-report.md) for what to fill in. For a problem that is not about one game, open a regular issue with the same zip. Built on Playden would not exist without these projects. Projects we carry patches for We keep a fork of each, with our changes on a feat/playden-ps5 branch, and send fixes upstream. | Project | What it does for Playden | | --- | --- | | prospero-win | Wine for the PS5. Runs the Windows games: memory management, a 32-bit x86 translator, display, input and audio for the console. Also builds the per-game app shell. | | ps5-homebrew-ui | The UI toolkit Playden's screens are drawn with: layout, fonts, controller focus and navigation, on OpenGL. | | ps5-opengl | OpenGL for PS5 homebrew. Draws Playden's own interface. | | ps5-native-app-boilerplate (mpereiraesaa) | Tools to build native PS5 apps and modules, used for the per-game app shell in the kit. Our fork is named prospero-foundation. | | PS5-Lapy-JB-Daemon | The root helper. Gives a game's app just enough rights to reach its own files in /data. | Used as they are | Project | What it does for Playden | | --- | --- | | ps5-native-app-boilerplate (blackbearreloaded) | Tools to build Playden as a native PS5 app, and its libc.prx. | | ps5-payload-sdk | The SDK for PS5 homebrew. | | gbe_fork | Steamworks emulator (LGPL-3.0), shipped with Playden in runtime/steamworks/, so games run without the Steam client. | Inside the runtime kit - Wine — runs Windows programs. - DXVK — turns Direct3D 8–11 into Vulkan. - PS5_Mesa and PS5_Vulkan — Mesa for the PS5: RADV, the AMD Vulkan driver, for the PS5 GPU, and Zink with WGL, which runs Windows OpenGL games on top of Vulkan. Small libraries curl, Mbed TLS, SQLite, zstd, LZMA SDK, stb, utf8proc and QR Code generator. Contributing Contributions are welcome. [CONTRIBUTING.md](CONTRIBUTING.md) explains the development setup, the tests, the console workflow, and how to change a forked dependency or a Wine patch. License Playden is licensed under the GNU General Public License v3.0 or later. See [LICENSE](LICENSE). The submodules, vendored libraries and the runtime kit keep their own licenses: see [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md). Steam is a trademark of Valve Corporation. PlayStation and PS5 are trademarks of Sony Interactive Entertainment. This project is not affiliated with either company.