[{"data":1,"prerenderedAt":7901},["ShallowReactive",2],{"latest-blog-posts":3},[4,283,558,868,1032,1345,1610,1884,2207,2587,2942,3132,3339,3464,3542,3648,4108,4593,4786,5311,6449,6967,7300,7513,7760],{"id":5,"title":6,"body":7,"category":269,"date":270,"description":21,"extension":271,"featured":272,"image":273,"meta":274,"navigation":272,"path":277,"readTime":278,"seo":279,"sitemap":280,"stem":281,"__hash__":282},"blog/blog/2026-04-27-the-hidden-costs-of-switching-from-batch-to-streaming.md","The Hidden Costs of Switching from Batch to Streaming",{"type":8,"value":9,"toc":254},"minimark",[10,17,22,25,30,33,36,39,42,45,49,52,55,58,61,64,67,70,74,77,80,83,86,89,96,100,103,106,109,113,116,119,122,125,129,132,135,138,142,145,148,151,155,158,161,164,167,174,178,181,184,187,190,193,195,199,202,205,208,210,214,217,220,223,230,232],[11,12,13],"p",{},[14,15,16],"em",{},"By Andrew Tan",[11,18,19],{},[14,20,21],{},"The expenses nobody puts in the migration budget — and why the real price of going real-time has nothing to do with software licenses",[23,24],"hr",{},[26,27,29],"h2",{"id":28},"the-budget-that-didnt-survive-first-contact","The budget that didn't survive first contact",[11,31,32],{},"A VP of Engineering I know budgeted $180,000 for his team's batch-to-streaming migration. That was twelve months ago. Last time we spoke, the project had consumed $640,000 and they were still six weeks from production.",[11,34,35],{},"What happened? Not fraud. Not scope creep in the traditional sense. They simply failed to account for the costs that don't appear in vendor quotes: the six-week delay while they hired a Kafka engineer who understood exactly-once semantics. The three months spent running batch and streaming in parallel because nobody trusted the new pipeline yet. The emergency consulting engagement when their streaming aggregation produced different numbers than the batch report and the CFO noticed.",[11,37,38],{},"The software itself was cheap. The hidden costs ate them alive.",[11,40,41],{},"I've watched this pattern repeat across companies of every size. Teams budget for infrastructure and licenses. They don't budget for uncertainty, rework, and the operational tax of maintaining two systems while one replaces the other. By the time they realize what's happening, the project is either over budget or under-delivered — sometimes both.",[11,43,44],{},"Here's what actually costs money when you move from batch to streaming.",[26,46,48],{"id":47},"cost-1-the-talent-you-dont-have-yet","Cost #1: The talent you don't have yet",[11,50,51],{},"Batch engineering and stream engineering are related the way carpentry and furniture making are related. Same raw material, completely different craft.",[11,53,54],{},"Your existing team knows cron schedules, table scans, and the comforting finality of a job that starts, runs, and finishes. Streaming asks them to think in event time, manage unbounded state, and debug systems that never stop running. Some of your engineers will adapt quickly. Others won't — not because they're bad engineers, but because distributed stream processing is genuinely difficult and not everyone wants to specialize in it.",[11,56,57],{},"This creates a hidden cost in three forms:",[11,59,60],{},"Hiring: A senior stream processing engineer in London or New York costs between $160,000 and $220,000 base salary right now, plus the four-month average time-to-hire for that specialty. If you need two of them, that's nearly half a million in salary before they've written a line of production code.",[11,62,63],{},"Training: Your existing engineers need to learn new concepts: watermarking, consumer lag, partition skew, stateful operators, at-least-once versus exactly-once. These aren't afternoon workshop topics. They're months of hands-on learning where productivity is lower than normal and mistakes are more expensive than usual.",[11,65,66],{},"Attrition: Some of your best batch engineers will leave during the migration — not because they can't learn streaming, but because they didn't sign up to become distributed systems specialists. They liked the data work. They'll go somewhere that still does it the way they enjoy.",[11,68,69],{},"The budget line item for \"talent\" in most migration plans covers training. It rarely covers hiring delays, lost productivity, or unexpected turnover.",[26,71,73],{"id":72},"cost-2-the-parallel-operation-period","Cost #2: The parallel operation period",[11,75,76],{},"Nobody talks about this enough. You cannot simply turn off batch and turn on streaming. Not if you value your job.",[11,78,79],{},"For some period — typically three to six months, occasionally longer — you'll run both systems. The batch pipeline keeps producing the reports everyone trusts. The streaming pipeline runs alongside it, producing results that theoretically should match but often don't, at least not at first.",[11,81,82],{},"This means double the infrastructure. Double the monitoring. Double the alerts. And a team of engineers spending their days reconciling two sets of numbers instead of building new features.",[11,84,85],{},"One e-commerce company I worked with ran parallel systems for eight months. Their batch stack cost roughly $4,200 per month in cloud compute. Their streaming stack cost $7,800 per month. For eight months, they paid both. That's $96,000 in infrastructure alone — never mind the engineering time spent investigating why the streaming count of Tuesday's orders was 347 off from the batch count.",[11,87,88],{},"The parallel period isn't optional. It's insurance. But like all insurance, it's expensive, and most teams underestimate the premium.",[11,90,91],{},[92,93],"img",{"alt":94,"src":95},"Engineers running batch and streaming systems side by side during migration","/images/blog/2026-04-27/parallel-systems.jpg",[26,97,99],{"id":98},"cost-3-the-data-archaeology","Cost #3: The data archaeology",[11,101,102],{},"Your batch pipelines contain years of accumulated business logic. Somewhere in a 400-line Python script that runs at 2 AM is a join condition that exists because of a pricing exception from 2019. Nobody documented why it's there. The person who wrote it left in 2021. But if you remove it, the revenue numbers shift by 0.3% and finance sends angry emails.",[11,104,105],{},"Migrating to streaming means understanding every one of these artifacts. You can't simply port the code. The logic needs to be reimplemented for continuous event processing, which means you first have to understand what it does and why. This is data archaeology — tedious, slow, and impossible to estimate accurately because you don't know what you'll find until you start digging.",[11,107,108],{},"A financial services firm I advised spent five weeks on a single pipeline. The streaming implementation took three days. Figuring out why the batch version produced a specific corner-case output took the other thirty-two days. The business logic was encoded in a stored procedure written by three different people over four years, with comments like \"fix for Q2 bug\" and no further explanation.",[26,110,112],{"id":111},"cost-4-the-operational-complexity-tax","Cost #4: The operational complexity tax",[11,114,115],{},"Batch pipelines fail visibly. A job crashes. You get an alert. You fix it. You rerun it. Everyone understands what happened.",[11,117,118],{},"Streaming pipelines fail subtly. Consumer lag builds over hours. State stores grow until they hit memory limits. Watermarks drift and suddenly your windowed aggregations are dropping late events. By the time you notice, you've been producing slightly wrong results for half a day.",[11,120,121],{},"The operational tooling is different too. You're not just monitoring whether a job finished. You're monitoring latency distributions, throughput slopes, backpressure signals, and state store sizes. Your existing runbooks don't apply. Your existing alerts don't catch the new failure modes.",[11,123,124],{},"Building this operational maturity takes time and mistakes. The first time your streaming pipeline silently drops 2% of events for six hours, you'll invest heavily in better observability. That's a necessary cost. But it's almost never in the initial budget.",[26,126,128],{"id":127},"cost-5-the-opportunity-cost-nobody-measures","Cost #5: The opportunity cost nobody measures",[11,130,131],{},"While your best engineers are debugging partition rebalancing and reconciling batch versus streaming outputs, they aren't doing other work. Feature requests pile up. Technical debt accumulates. Competitors ship things your team would have built if they weren't neck-deep in migration.",[11,133,134],{},"This is the hardest cost to quantify and the easiest to ignore. There's no invoice for it. But it's real.",[11,136,137],{},"One SaaS company paused all new data product development for nine months during their streaming migration. When they finished, they'd built a technically impressive real-time pipeline — but their primary competitor had shipped three analytics features in the same period and gained market share. The migration was a technical success and a strategic delay.",[26,139,141],{"id":140},"why-we-keep-underestimating","Why we keep underestimating",[11,143,144],{},"Part of the problem is vendor messaging. Streaming platforms sell the destination: real-time insights, instant reactions, competitive advantage. They don't advertise the journey: the hiring, the parallel systems, the archaeology, the operational learning curve.",[11,146,147],{},"Another part is optimism bias. Every engineering team believes they'll be the exception. Their code is cleaner. Their team is smarter. Their requirements are simpler. Sometimes that's true. Usually it isn't.",[11,149,150],{},"The result is a persistent gap between the budgeted cost and the actual cost. I've seen ratios of 2:1, 3:1, even 5:1. Not because anyone was dishonest — because the real costs are invisible until you've already committed.",[26,152,154],{"id":153},"how-to-budget-honestly","How to budget honestly",[11,156,157],{},"You can't eliminate these costs, but you can account for them. Here's how I advise teams to think about it:",[11,159,160],{},"Add a 40% buffer to infrastructure estimates. The parallel period, the testing environments, the shadow deployments — they all add compute and storage you won't predict precisely.",[11,162,163],{},"Budget for six months of dual operation minimum. If you finish earlier, celebrate. If you don't, you won't be explaining overruns to your CFO.",[11,165,166],{},"Hire or contract one streaming specialist before you start, not after you get stuck. The cost of bringing them in early is high. The cost of bringing them in after three months of false starts is higher.",[11,168,169,170,173],{},"Accept that some pipelines should stay batch. Not everything benefits from real-time. Daily reporting, historical analytics, ML training pipelines — these are often batch-appropriate workloads that don't justify the migration cost. Be explicit about what you're ",[14,171,172],{},"not"," migrating.",[26,175,177],{"id":176},"a-different-way-to-think-about-the-transition","A different way to think about the transition",[11,179,180],{},"The teams that handle this well share one trait: they don't view it as a migration. They view it as adding a capability.",[11,182,183],{},"Instead of \"we're moving from batch to streaming,\" they say \"we're adding streaming where it creates value, and keeping batch where it still works.\" This sounds like semantics, but it changes the economics completely. You're no longer committed to moving everything. You can evaluate each pipeline on its own merits: latency requirements, complexity, business value, migration cost.",[11,185,186],{},"Some pipelines move. Some don't. The ones that move justify their own investment. The ones that stay don't generate unnecessary cost.",[11,188,189],{},"This is where a unified platform matters. If you're running separate tools for batch and streaming, every pipeline faces pressure to migrate because maintaining two platforms is expensive. If you can run both models on the same platform — same workflows, same team, same operational approach — the pressure disappears. You add streaming where it earns its keep and leave batch alone where it's already working.",[11,191,192],{},"That's the approach we built into layline.io. Not because batch is bad — it's often exactly right — but because forcing teams to choose one approach and abandon the other creates artificial cost and risk. The teams that sleep well at night are the ones that didn't try to boil the ocean.",[23,194],{},[26,196,198],{"id":197},"the-bottom-line","The bottom line",[11,200,201],{},"The hidden cost of switching from batch to streaming isn't the software. It's the everything else: the people you need to hire, the systems you need to run in parallel, the legacy logic you need to excavate, the operational maturity you need to build, and the opportunity you lose while you're focused on infrastructure instead of product.",[11,203,204],{},"Budget for it. Account for it. Be honest about which pipelines actually need to move and which don't.",[11,206,207],{},"The goal isn't to be real-time everywhere. The goal is to be real-time where it matters, without bankrupting yourself to get there.",[23,209],{},[26,211,213],{"id":212},"whats-next","What's next",[11,215,216],{},"If you're planning a batch-to-streaming migration, start with an honest audit. List your top ten pipelines. For each one, ask: what's the actual cost of latency? What's the estimated migration effort? What's the operational complexity add?",[11,218,219],{},"If the numbers don't justify the move for a given pipeline, leave it alone. Focus your energy on the two or three where real-time creates measurable business value.",[11,221,222],{},"For teams evaluating platforms, the Community Edition of layline.io is free to explore. You can prototype a streaming pipeline alongside your existing batch workflow and see what the operational reality looks like before you commit the budget.",[11,224,225],{},[226,227,229],"a",{"href":228},"/get-started","Try the Community Edition →",[23,231],{},[233,234,236,237,236,242],"div",{"style":235},"display: flex; align-items: center; gap: 1rem; margin-top: 2rem;","\n  ",[92,238],{"src":239,"alt":240,"style":241},"/images/blog/authors/andrew-tan.jpeg","Andrew Tan","width: 80px; height: 80px; border-radius: 50%; object-fit: cover; flex-shrink: 0;",[11,243,245,248,249,253],{"style":244},"margin: 0;",[246,247,240],"strong",{}," is a serial entrepreneur and founder of ",[226,250,252],{"href":251},"https://layline.io","layline.io",", building enterprise data processing infrastructure that handles both batch and real-time workloads at scale.",{"title":255,"searchDepth":256,"depth":256,"links":257},"",2,[258,259,260,261,262,263,264,265,266,267,268],{"id":28,"depth":256,"text":29},{"id":47,"depth":256,"text":48},{"id":72,"depth":256,"text":73},{"id":98,"depth":256,"text":99},{"id":111,"depth":256,"text":112},{"id":127,"depth":256,"text":128},{"id":140,"depth":256,"text":141},{"id":153,"depth":256,"text":154},{"id":176,"depth":256,"text":177},{"id":197,"depth":256,"text":198},{"id":212,"depth":256,"text":213},"Article","2026-04-27","md",true,"/images/blog/2026-04-27/hidden-costs-hero.jpg",{"author":275},{"name":240,"image":239,"url":276},"https://www.linkedin.com/in/andrewtan/","/blog/2026-04-27-the-hidden-costs-of-switching-from-batch-to-streaming","9 min",{"title":6,"description":21},{"loc":277},"blog/2026-04-27-the-hidden-costs-of-switching-from-batch-to-streaming","rpmdUkvZ30Y4orxnZqgZVQPy30Kq_giPM6z2wXWENfQ",{"id":284,"title":285,"body":286,"category":269,"date":548,"description":296,"extension":271,"featured":272,"image":549,"meta":550,"navigation":272,"path":552,"readTime":553,"seo":554,"sitemap":555,"stem":556,"__hash__":557},"blog/blog/2026-04-20-financial-data-integration.md","Financial Data Integration: A Practical Guide",{"type":8,"value":287,"toc":533},[288,292,297,299,303,306,309,312,316,319,322,325,328,331,335,338,343,346,349,355,358,374,377,380,384,387,390,393,407,410,413,417,420,426,429,432,436,439,442,445,448,451,455,458,472,475,479,482,485,488,491,494,497,499,501,504,507,510,512,514,517,520,523],[11,289,290],{},[14,291,16],{},[11,293,294],{},[14,295,296],{},"Why integrating financial data is uniquely difficult, what makes it different from regular ETL, and how teams actually solve it without breaking everything",[23,298],{},[26,300,302],{"id":301},"the-compliance-problem-nobody-talks-about","The compliance problem nobody talks about",[11,304,305],{},"At a typical mid-size bank, a data integration project gets delayed for months. Not because of technical problems. Not because of budget. Because nobody can agree on what \"the single source of truth\" actually means.",[11,307,308],{},"The trading desk has one definition. Risk management has another. Regulatory reporting needs a third. Each team has built their own pipelines over the years — some in Python, some in SQL stored procedures, one terrifying COBOL script that nobody dares touch. Getting them to agree on unified data models feels like negotiating a peace treaty.",[11,310,311],{},"This is financial data integration in a nutshell. It's not just about moving data from A to B. It's about reconciling decades of accumulated business logic, dealing with regulatory minefields, and somehow making it all work in real-time without taking down systems that process billions in transactions daily.",[26,313,315],{"id":314},"why-financial-data-is-different","Why financial data is different",[11,317,318],{},"Most ETL articles assume you're working with relatively clean data in modern formats, processed in batches overnight. Financial services breaks every one of those assumptions.",[11,320,321],{},"The data formats are ancient and proprietary. While the rest of the world moved to JSON and REST APIs, financial services still runs on FIX protocol, SWIFT messages, ISO 20022 XML, and a dizzying array of vendor-specific binary formats. A single trading firm might receive market data in one format, execute orders in another, and settle trades in a third — all for the same transaction.",[11,323,324],{},"Latency requirements are brutal. In high-frequency trading, microseconds matter. A retail bank's fraud detection system needs to score transactions in under 100 milliseconds or customers get annoyed waiting for their card to work. Traditional batch ETL, with its hourly or daily windows, simply doesn't work here.",[11,326,327],{},"Regulatory requirements are non-negotiable. MiFID II in Europe requires trade reporting within minutes. Basel III demands real-time risk calculations. GDPR means you need to track exactly where personal data flows and be able to delete it on request. Get this wrong and you're not just debugging a pipeline — you're explaining yourself to regulators.",[11,329,330],{},"The stakes are higher. A failed ETL job at an e-commerce company means delayed reports. A failed pipeline at a bank can mean failed trades, regulatory breaches, or incorrect risk exposure calculations. Recovery time objectives are measured in seconds, not hours.",[26,332,334],{"id":333},"the-three-integration-patterns-that-actually-work","The three integration patterns that actually work",[11,336,337],{},"Across the financial services industry, three approaches consistently succeed. The key is matching the pattern to your actual constraints, not what you'd prefer them to be.",[339,340,342],"h3",{"id":341},"pattern-1-the-event-driven-backbone","Pattern 1: The event-driven backbone",[11,344,345],{},"This is becoming the standard for modern financial infrastructure. Instead of polling databases every few minutes, you stream events as they happen.",[11,347,348],{},"A trade executes? That's an event. A payment clears? Another event. Risk thresholds breached? Event. Each system subscribes to the events it cares about and reacts in real-time.",[11,350,351],{},[92,352],{"alt":353,"src":354},"Event-driven architecture with CDC, Kafka, and stream processors","/images/blog/2026-04-20/event-driven-backbone.jpg",[11,356,357],{},"The architecture usually looks like this:",[359,360,361,365,368,371],"ul",{},[362,363,364],"li",{},"CDC (Change Data Capture) connectors watch legacy databases and emit events when rows change",[362,366,367],{},"Kafka or similar is the central nervous system, durably storing events",[362,369,370],{},"Stream processors handle transformations, aggregations, and routing",[362,372,373],{},"Target systems consume exactly what they need, when they need it",[11,375,376],{},"Many fintechs use this pattern to connect modern microservices with legacy mainframes. The mainframe continues running the core ledger (too risky to migrate), but CDC connectors stream every transaction change to Kafka within milliseconds. New services build on this event stream without ever touching the legacy database directly.",[11,378,379],{},"The downside? Event-driven systems are harder to reason about than batch jobs. When something goes wrong, you can't just \"re-run yesterday's job.\" You need to understand the event topology, replay strategies, and exactly-once semantics.",[339,381,383],{"id":382},"pattern-2-the-api-gateway-layer","Pattern 2: The API gateway layer",[11,385,386],{},"For teams dealing with external data sources — market data feeds, counterparty APIs, regulatory reporting services — an API gateway pattern often works better than pure streaming.",[11,388,389],{},"The idea is simple: create a unified abstraction layer that normalizes all those different data sources into a consistent internal format. Your trading systems don't need to know that Bloomberg speaks one protocol and Refinitiv speaks another. They just call your internal API.",[11,391,392],{},"This pattern shines when:",[359,394,395,398,401,404],{},[362,396,397],{},"You're integrating with many external vendors who each have their own quirks",[362,399,400],{},"You need to cache and fan-out data to multiple internal consumers",[362,402,403],{},"You want to enforce security, rate limiting, and audit logging in one place",[362,405,406],{},"You need to switch vendors without rewriting downstream systems",[11,408,409],{},"Wealth management firms often use this approach for market data. They normalize feeds from multiple providers into a single internal format, add real-time validation and entitlements, then expose it via GraphQL or REST. Portfolio managers get exactly the data they need, formatted consistently, regardless of which vendor supplied the underlying feed.",[11,411,412],{},"The catch is operational complexity. You're now running a critical piece of infrastructure that everything depends on. When the gateway has issues, everything has issues.",[339,414,416],{"id":415},"pattern-3-the-hybrid-compromise","Pattern 3: The hybrid compromise",[11,418,419],{},"Most mature financial institutions end up here. You keep batch processing for the workloads that genuinely don't need real-time — regulatory reports, end-of-day reconciliation, historical analytics. You add streaming for the latency-sensitive workflows — fraud detection, risk monitoring, customer-facing dashboards.",[11,421,422],{},[92,423],{"alt":424,"src":425},"Hybrid batch and streaming architecture","/images/blog/2026-04-20/hybrid-architecture.jpg",[11,427,428],{},"The key is being intentional about the boundary. Not everything needs to be real-time, and trying to force streaming on batch-appropriate workloads just creates unnecessary complexity.",[11,430,431],{},"Trading platforms typically keep overnight risk calculations in batch (the math is complex and doesn't need to be instant), but move position monitoring to streaming (traders need to know their exposure immediately). The two systems coexist, with the streaming layer feeding into the batch layer for end-of-day reconciliation.",[26,433,435],{"id":434},"the-hidden-challenges-nobody-talks-about","The hidden challenges nobody talks about",[11,437,438],{},"Beyond the architectural patterns, there are specific problems that catch teams off guard.",[11,440,441],{},"Reference data is a nightmare. Every trade references securities, counterparties, and market identifiers that exist in master data systems. Those master systems update on their own schedules. If your trade data references a security that hasn't been loaded into your local cache yet, what happens? Financial data integration requires sophisticated reference data management — caching strategies, fallback logic, and tolerance for temporarily incomplete data.",[11,443,444],{},"Time zones and market hours. A global trading operation spans Tokyo, London, and New York. Each market opens and closes at different times. Some instruments trade 24/7. Your data pipelines need to handle \"end of day\" concepts that vary by instrument, geography, and market regime. The simple notion of \"yesterday's data\" becomes surprisingly complex.",[11,446,447],{},"Data quality at scale. When you're processing millions of transactions per hour, even 0.01% bad data is hundreds of errors to investigate. Financial data integration requires automated quality checks — schema validation, range checks, referential integrity — that can run in real-time and route suspicious data to human review queues without blocking the pipeline.",[11,449,450],{},"Testing in production. You can't exactly spin up a copy of a global trading system to test your new pipeline. Teams often use techniques like shadow mode (run new and old pipelines in parallel, compare outputs) or synthetic transactions (inject test trades that get processed but not settled) to validate changes.",[26,452,454],{"id":453},"what-good-looks-like","What good looks like",[11,456,457],{},"When financial data integration works, you notice it in the operational metrics:",[359,459,460,463,466,469],{},[362,461,462],{},"Reconciliation exceptions drop. When data flows consistently across systems, the daily \"why don't these numbers match\" investigations become rare.",[362,464,465],{},"Time-to-insight shrinks. A risk manager can see their current exposure without waiting for the overnight batch. A compliance officer can generate regulatory reports on demand, not on schedule.",[362,467,468],{},"System outages become isolated. When one system has issues, it doesn't cascade through brittle batch dependencies.",[362,470,471],{},"New projects move faster. Teams spend less time figuring out how to get data and more time using it.",[11,473,474],{},"But getting there requires more than technology. It requires organizational agreement on data ownership, quality standards, and change management processes. The technical solution is often the easy part.",[26,476,478],{"id":477},"where-laylineio-fits-in","Where layline.io fits in",[11,480,481],{},"If you're evaluating platforms for financial data integration, here's where layline.io is worth considering:",[11,483,484],{},"It handles both batch and streaming in the same platform. This matters because most financial institutions need both — and having separate tools for each creates unnecessary complexity and context switching.",[11,486,487],{},"The visual workflow designer helps with the organizational challenge. When compliance, trading, and IT teams can all see and understand the data flows, agreement becomes easier. You spend less time in meetings explaining what the pipeline does and more time improving it.",[11,489,490],{},"It includes built-in handling for the operational concerns that matter in finance: exactly-once processing guarantees, stateful operations with checkpointing, backpressure management when downstream systems slow down. These aren't afterthoughts — they're core features.",[11,492,493],{},"The infrastructure-agnostic deployment means you can run it where your compliance team is comfortable: on-premises, in your existing cloud environment, or air-gapped if that's what your security requirements demand.",[11,495,496],{},"For teams that need financial-grade data integration without building a dedicated platform engineering team, this is the gap it fills.",[23,498],{},[26,500,198],{"id":197},[11,502,503],{},"Financial data integration is harder than regular ETL because the constraints are tighter, the stakes are higher, and the systems you're integrating are older and more complex. But the patterns that work are well understood: event-driven architectures for real-time needs, API gateways for external integration, and hybrid approaches that don't force streaming on batch-appropriate workloads.",[11,505,506],{},"The teams that succeed focus first on understanding their actual requirements — latency needs, regulatory constraints, data quality standards — before choosing technology. They invest in reference data management and testing strategies that work at financial scale. And they accept that some problems are organizational, not technical.",[11,508,509],{},"Start with one high-value pipeline. Prove the pattern. Then expand. Whether you build it yourself or use a platform like layline.io, the key is being intentional about where real-time actually matters and where batch is still the right answer.",[23,511],{},[26,513,213],{"id":212},[11,515,516],{},"If you're wrestling with financial data integration, the best next step is mapping your actual data flows. Not the architecture diagrams — the real flows, including the Excel exports, the email attachments, and the scripts that run on Bob's desktop because nobody else knows how they work.",[11,518,519],{},"Once you see the full picture, you can identify which integrations would benefit most from modernization. Start there.",[11,521,522],{},"For layline.io users, the Community Edition is free to try — no credit card required. You can prototype a streaming pipeline against your existing data sources and see how it handles your specific formats and requirements.",[233,524,236,525,236,527],{"style":235},[92,526],{"src":239,"alt":240,"style":241},[11,528,529,248,531,253],{"style":244},[246,530,240],{},[226,532,252],{"href":251},{"title":255,"searchDepth":256,"depth":256,"links":534},[535,536,537,543,544,545,546,547],{"id":301,"depth":256,"text":302},{"id":314,"depth":256,"text":315},{"id":333,"depth":256,"text":334,"children":538},[539,541,542],{"id":341,"depth":540,"text":342},3,{"id":382,"depth":540,"text":383},{"id":415,"depth":540,"text":416},{"id":434,"depth":256,"text":435},{"id":453,"depth":256,"text":454},{"id":477,"depth":256,"text":478},{"id":197,"depth":256,"text":198},{"id":212,"depth":256,"text":213},"2026-04-20","/images/blog/2026-04-20/financial-data-integration-hero.jpg",{"author":551},{"name":240,"image":239,"url":276},"/blog/2026-04-20-financial-data-integration","8 min",{"title":285,"description":296},{"loc":552},"blog/2026-04-20-financial-data-integration","XjeoZl7mN59uwTM8Tvzr8eH-bxnrfEw174bJtvMbDjk",{"id":559,"title":560,"body":561,"category":269,"date":859,"description":571,"extension":271,"featured":272,"image":860,"meta":861,"navigation":272,"path":863,"readTime":278,"seo":864,"sitemap":865,"stem":866,"__hash__":867},"blog/blog/2026-04-17-layline-vs-kafka.md","layline.io vs. Apache Kafka: When to Choose What",{"type":8,"value":562,"toc":840},[563,567,572,574,578,581,584,587,590,594,597,600,603,633,637,640,643,646,649,655,659,662,665,668,672,675,678,682,685,688,691,695,698,701,705,708,712,729,733,750,754,762,765,771,775,778,781,784,787,791,794,797,800,803,805,808,811,814,816,818,821,824,828,830],[11,564,565],{},[14,566,16],{},[11,568,569],{},[14,570,571],{},"A practical comparison of stream processing approaches — covering latency, operational complexity, and the team fit that actually determines the right choice",[23,573],{},[26,575,577],{"id":576},"the-meeting-that-wouldnt-end","The meeting that wouldn't end",[11,579,580],{},"I sat in a conference room last year with a team that had been debating Kafka for months. Not debating whether to stream data. That decision was made. They were debating whether their team could actually operate Kafka in production without hiring three infrastructure engineers they couldn't afford.",[11,582,583],{},"The architect loved Kafka. He'd used it at a previous company and knew what it could do. The engineering manager was skeptical. She'd read the post-mortems from teams that spent quarters tuning consumer groups and still couldn't get exactly-once semantics right. The CTO just wanted to ship. The project was already behind schedule.",[11,585,586],{},"By hour three, they'd agreed on nothing except that they needed lunch.",[11,588,589],{},"This is the Kafka decision in a nutshell. It's not a technology problem. It's a fit problem. Kafka is the right answer more often than people admit. It's also the wrong answer more often than people admit. The difference isn't in the feature matrix. It's in what your team is actually good at, what your workload actually needs, and what you're willing to own operationally.",[26,591,593],{"id":592},"what-kafka-is-actually-for","What Kafka is actually for",[11,595,596],{},"Let's start with what Kafka does exceptionally well, because too many comparisons skip this part:",[11,598,599],{},"Kafka is a distributed event log. Its core superpower is durability at scale. You can pump millions of events per second into Kafka, spread them across a cluster, and read them back in order from multiple consumers. It doesn't care if your consumers are fast or slow. It doesn't care if they crash and restart. The events stay there until they expire.",[11,601,602],{},"This makes Kafka the right choice when:",[604,605,606,617,627],"ol",{},[362,607,608,611,612,615],{},[246,609,610],{},"You need a central nervous system:"," Multiple teams need to consume the same events. Marketing needs clickstreams. Analytics needs aggregates. Operations needs alerts. Kafka decouples producers from consumers so each team can read at their own pace without coordinating deployments.",[613,614],"br",{},[613,616],{},[362,618,619,622,623,625],{},[246,620,621],{},"Durability matters more than latency:"," Kafka isn't the fastest message broker. It's fast enough for most use cases, but if you're doing high-frequency trading where microseconds matter, you'll look elsewhere. Where Kafka shines is guaranteeing that an event, once acknowledged, will survive multiple disk failures and node crashes.",[613,624],{},[613,626],{},[362,628,629,632],{},[246,630,631],{},"Your team already knows distributed systems:"," Kafka is not a managed service you forget about. Even with managed offerings like Confluent Cloud, you still need people who understand partition rebalancing, consumer group coordination, offset management, and the subtle ways replication can fail. If you have those people, Kafka is a force multiplier. If you don't, it's a time sink.",[26,634,636],{"id":635},"where-kafka-gets-expensive","Where Kafka gets expensive",[11,638,639],{},"The hidden cost of Kafka isn't the licensing. It's the operational expertise.",[11,641,642],{},"I've talked to teams that spent nine months getting Kafka stable in production. Not because the software is bad — it's excellent — but because the operational surface area is enormous. You need to monitor lag, balance partitions, tune batch sizes, manage schema evolution, and debug consumer rebalances at 2 AM. These aren't one-time setup tasks. They're ongoing operational responsibilities.",[11,644,645],{},"The stream processing layer adds another dimension. Kafka itself is an event log. If you want to transform, aggregate, or join those streams, you need a stream processor: Kafka Streams, Flink, ksqlDB, or Spark Streaming. Each of these is a significant technology in its own right. You're not just operating Kafka. You're operating a streaming stack.",[11,647,648],{},"This is where the decision gets painful for smaller teams. They want real-time processing. They need event-driven architecture. But they don't have a platform engineering team to babysit a Kafka cluster and a Flink job cluster. They have five backend engineers who also maintain the API and the database.",[11,650,651],{},[92,652],{"alt":653,"src":654},"The operational complexity of managing a Kafka streaming stack","/images/blog/2026-04-17/kafka-operational-complexity.jpg",[26,656,658],{"id":657},"what-laylineio-does-differently","What layline.io does differently",[11,660,661],{},"We built layline.io for teams in exactly that situation. Not because Kafka is bad, but because the full Kafka + stream processor stack is overkill for a lot of workloads — and under-resourced for the teams that choose it.",[11,663,664],{},"layline.io is a unified data processing platform. It handles both batch and streaming workloads with the same workflows, the same visual designer, and the same operational model. You don't need separate tools for batch ETL and real-time streaming. You don't need separate teams with separate expertise.",[11,666,667],{},"The key differences come down to three things:",[339,669,671],{"id":670},"_1-operational-abstraction","1. Operational abstraction",[11,673,674],{},"With Kafka, you're operating infrastructure. With layline.io, you're operating workflows. The platform handles partitioning, state management, checkpointing, and backpressure automatically. You design your pipeline visually, deploy it, and monitor it through the same interface. The operational surface area is much smaller.",[11,676,677],{},"This doesn't mean layline.io is \"Kafka without the complexity.\" Under the hood, the engine handles many of the same distributed systems problems. The difference is that you don't have to handle them yourself. For teams without dedicated infrastructure engineers, that's the difference between shipping in weeks and shipping in quarters.",[339,679,681],{"id":680},"_2-unified-batch-and-streaming","2. Unified batch and streaming",[11,683,684],{},"Most real-world environments need both. You need real-time fraud detection. You also need end-of-day reconciliation reports. You need streaming alerts. You also need monthly analytics exports.",[11,686,687],{},"With a Kafka-centric stack, you typically end up with two separate systems: Kafka + Flink for streaming, and Airflow or dbt for batch. Two codebases. Two operational models. Two sets of expertise.",[11,689,690],{},"layline.io runs both on the same platform. The same workflow can process a batch file or a streaming topic. The same team can build and operate both. For organizations that aren't large enough to justify separate streaming and batch teams, this is a significant simplification.",[339,692,694],{"id":693},"_3-visual-workflow-design","3. Visual workflow design",[11,696,697],{},"This sounds like a feature, but it's actually a collaboration issue. When your data pipeline is written in Java or Scala and lives in a Git repo, only the engineers who wrote it can change it. Business analysts, data scientists, and operations teams are blocked.",[11,699,700],{},"layline.io's visual workflow designer makes the data flow explicit. Non-engineers can read it. Engineers can modify it without hunting through thousands of lines of stream processing code. In practice, this means fewer miscommunications between the people who understand the business logic and the people who maintain the infrastructure.",[26,702,704],{"id":703},"the-decision-framework","The decision framework",[11,706,707],{},"Here's how I think about the choice in practice.",[339,709,711],{"id":710},"choose-kafka-when","Choose Kafka when",[359,713,714,717,720,723,726],{},[362,715,716],{},"You need a company-wide event bus that multiple teams consume independently",[362,718,719],{},"You have (or can hire) engineers with deep Kafka operational experience",[362,721,722],{},"You already run a separate batch stack and don't mind maintaining both",[362,724,725],{},"Your workload is primarily event streaming with relatively simple transformations",[362,727,728],{},"Durability and decoupling are more important than time-to-production",[339,730,732],{"id":731},"choose-laylineio-when","Choose layline.io when",[359,734,735,738,741,744,747],{},[362,736,737],{},"You need both batch and streaming and want one platform for both",[362,739,740],{},"Your team is small and can't dedicate engineers to infrastructure operations",[362,742,743],{},"Your pipelines involve complex transformations, enrichment, and routing",[362,745,746],{},"You need business and technical teams to collaborate on pipeline design",[362,748,749],{},"Time-to-production and operational simplicity matter as much as raw throughput",[339,751,753],{"id":752},"use-both-together-when","Use both together when",[359,755,756,759],{},[362,757,758],{},"Kafka is already your central event log, but you need a more accessible layer for building workflows on top of it",[362,760,761],{},"You want to keep Kafka as the durable message bus while using layline.io for the complex stream processing, transformations, and batch orchestration",[11,763,764],{},"This hybrid pattern is more common than people think. Kafka is excellent at moving events durably. layline.io is excellent at processing them. The two complement each other cleanly.",[11,766,767],{},[92,768],{"alt":769,"src":770},"Choosing the right technology path for your team and workload","/images/blog/2026-04-17/technology-decision-crossroads.jpg",[26,772,774],{"id":773},"a-real-world-example","A real-world example",[11,776,777],{},"A mid-sized franchise we worked with had this exact decision. They were extending their fraud detection to real-time. The events came from payment processors, needed enrichment from customer databases, and had to trigger risk scoring within 200 milliseconds.",[11,779,780],{},"Their initial plan was Kafka + Flink. The architecture looked clean on the whiteboard. But after three months, they realized they were spending 80% of their time tuning Flink checkpointing and debugging Kafka consumer lag, and 20% of their time on the actual fraud logic.",[11,782,783],{},"They switched to a hybrid approach. Kafka remained the event log — it was already integrated with their payment processors. layline.io handled the enrichment, scoring, and alerting workflows. The team went from spending most of their time on infrastructure to spending most of their time on fraud models.",[11,785,786],{},"The interesting part? Their latency didn't increase. In some cases it decreased, because they weren't fighting operational fires that added unpredictability. What changed was where their engineering effort went.",[26,788,790],{"id":789},"the-mistake-most-teams-make","The mistake most teams make",[11,792,793],{},"The biggest mistake I see is choosing technology based on a benchmark or a feature list rather than team fit.",[11,795,796],{},"Kafka will beat layline.io on raw throughput in a benchmark. If your only criterion is events per second, Kafka wins. But raw throughput isn't what determines project success. What determines success is whether your team can build, operate, and evolve the system in production over multiple years.",[11,798,799],{},"I've seen teams choose Kafka because \"Netflix uses it\" and then struggle because they don't have Netflix's platform engineering organization. I've seen teams choose lighter-weight tools because they were easier to learn, then hit walls when they needed enterprise-grade durability.",[11,801,802],{},"The right question isn't \"which one is better?\" The right question is \"which one is better for us, given our team, our constraints, and our timeline?\"",[26,804,198],{"id":197},[11,806,807],{},"Kafka is a brilliant piece of engineering. For the right team and the right workload, it's unmatched. But it's not a universal answer, and pretending it is has cost a lot of teams a lot of sleepless nights.",[11,809,810],{},"layline.io exists because there's a large middle ground of teams that need real-time data processing but can't justify the operational overhead of a full Kafka + Flink stack. They need the results of stream processing without needing to become distributed systems experts.",[11,812,813],{},"Neither tool is a silver bullet. Both are excellent at what they're designed for. The art is knowing which one matches your reality.",[23,815],{},[26,817,213],{"id":212},[11,819,820],{},"If you're evaluating stream processing platforms, the best next step is a simple audit. List your top three use cases. Estimate the latency requirements. Be honest about your team's operational bandwidth. Then test the candidates against your actual workloads, not a benchmark someone else ran.",[11,822,823],{},"If you want to see how layline.io handles real-time and batch workloads on the same platform, the Community Edition is free to explore. You can build a prototype against your existing Kafka topics or data sources and compare the operational experience directly.",[11,825,826],{},[226,827,229],{"href":228},[23,829],{},[233,831,236,832,236,834],{"style":235},[92,833],{"src":239,"alt":240,"style":241},[11,835,836,248,838,253],{"style":244},[246,837,240],{},[226,839,252],{"href":251},{"title":255,"searchDepth":256,"depth":256,"links":841},[842,843,844,845,850,855,856,857,858],{"id":576,"depth":256,"text":577},{"id":592,"depth":256,"text":593},{"id":635,"depth":256,"text":636},{"id":657,"depth":256,"text":658,"children":846},[847,848,849],{"id":670,"depth":540,"text":671},{"id":680,"depth":540,"text":681},{"id":693,"depth":540,"text":694},{"id":703,"depth":256,"text":704,"children":851},[852,853,854],{"id":710,"depth":540,"text":711},{"id":731,"depth":540,"text":732},{"id":752,"depth":540,"text":753},{"id":773,"depth":256,"text":774},{"id":789,"depth":256,"text":790},{"id":197,"depth":256,"text":198},{"id":212,"depth":256,"text":213},"2026-04-17","/images/blog/2026-04-17/kafka-comparison-hero.jpg",{"author":862},{"name":240,"image":239,"url":276},"/blog/2026-04-17-layline-vs-kafka",{"title":560,"description":571},{"loc":863},"blog/2026-04-17-layline-vs-kafka","chzkGaewwxw_AOw-cHVhEG8AVqV7iEDVr_NW5YAGvP4",{"id":869,"title":870,"body":871,"category":269,"date":1022,"description":881,"extension":271,"featured":272,"image":1023,"meta":1024,"navigation":272,"path":1026,"readTime":1027,"seo":1028,"sitemap":1029,"stem":1030,"__hash__":1031},"blog/blog/2026-04-13-why-real-time-data-integration-matters.md","Why Real-Time Data Integration Matters for Modern Applications",{"type":8,"value":872,"toc":1014},[873,877,882,884,888,891,894,897,906,910,913,916,919,922,925,929,932,935,938,947,951,954,957,960,963,966,968,971,974,977,980,983,986,990,993,996,999,1002,1004],[11,874,875],{},[14,876,16],{},[11,878,879],{},[14,880,881],{},"The difference between near-real-time and actually-real-time, and why the gap costs more than you think",[23,883],{},[26,885,887],{"id":886},"the-47-million-delay","The €4.7 million delay",[11,889,890],{},"A major European retailer lost €4.7 million on Black Friday 2024. Not because their website crashed. Not because they ran out of stock. Because their \"real-time\" inventory system was running four hours behind.",[11,892,893],{},"340,000 customers placed orders for items that had already sold out. The system showed availability. The warehouse had none. By the time the discrepancy surfaced, the damage was done. Refunds issued. Customer service overwhelmed. Brand reputation dented. The post-mortem revealed something awkward: the pipeline was never designed for real-time. It was designed for \"near-real-time,\" a distinction that sounded technical in architecture reviews and turned out to be catastrophic in production.",[11,895,896],{},"I've heard versions of this story dozens of times. The gap between what \"real-time\" promises and what most systems deliver is wider than most teams realize. And it's getting wider, not narrower, as customer expectations accelerate.",[11,898,899,903],{},[92,900],{"alt":901,"src":902},"Formula 1 pit crew synchronizing data streams in real-time","/images/blog/2026-04-13/pit-stop-data-streams.jpg",[14,904,905],{},"Like a Formula 1 pit stop, real-time data processing requires precision, coordination, and the right infrastructure.",[26,907,909],{"id":908},"what-real-time-actually-means-and-doesnt","What \"real-time\" actually means (and doesn't)",[11,911,912],{},"The industry has muddied this water. Three categories get conflated under the same label.",[11,914,915],{},"Batch means hours or days between updates. Your nightly ETL job. Your weekly report. Clear boundaries, predictable windows, well-understood failure modes.",[11,917,918],{},"Near-real-time means minutes between updates. The system checks every five, fifteen, thirty minutes. Most \"real-time dashboards\" fall here. Good for many use cases. Not good for the ones that matter most.",[11,920,921],{},"Real-time means seconds or sub-second. The event happens. The system knows. The downstream action triggers immediately.",[11,923,924],{},"The retailer didn't have a real-time problem. They had a near-real-time system marketed as real-time, and nobody questioned the difference until it cost them four million euros.",[26,926,928],{"id":927},"three-forces-driving-the-shift","Three forces driving the shift",[11,930,931],{},"The Amazon effect. Customers expect instant everything. Not because they analyzed the technical requirements. Because that's what they've been trained to expect. A 2022 Shopify study of 12,000 consumers found 73% expect checkout, inventory, and shipping updates in real time. Not \"within the hour.\" Real time.",[11,933,934],{},"Operational windows are shrinking. Fraud detection after the transaction isn't detection. It's notification. The money's already gone. Manufacturing lines that wait for batch quality reports produce bad units for hours before someone notices. The cost of delay compounds faster than most spreadsheets capture.",[11,936,937],{},"Competitive pressure. If your competitor updates pricing every thirty seconds and you update every six hours, you're not competing. You're spectating. This isn't theoretical. E-commerce platforms, travel aggregators, financial services. The companies winning in these spaces made real-time data infrastructure a strategic priority, not a technical nice-to-have.",[11,939,940,944],{},[92,941],{"alt":942,"src":943},"Formula 1 race car leaving a trail of streaming data","/images/blog/2026-04-13/racing-car-data-trail.jpg",[14,945,946],{},"Speed without control is dangerous. Real-time systems need to handle velocity while maintaining accuracy and reliability.",[26,948,950],{"id":949},"the-hidden-complexity","The hidden complexity",[11,952,953],{},"Moving from batch to streaming is harder than it looks. The surface seems simple: instead of waiting, react immediately. Underneath, everything changes.",[11,955,956],{},"State management. Batch jobs process bounded datasets. You know the input size when you start. Streaming processes unbounded streams. You need to track windows, handle late-arriving data, manage state across events that may arrive out of order.",[11,958,959],{},"Exactly-once processing. Run a batch job twice by accident? You get duplicate output, fix it, move on. Run a streaming pipeline twice? You double-charge customers, double-count inventory, double-notify systems. The semantics matter in ways they didn't before.",[11,961,962],{},"Backpressure. What happens when your source produces faster than your sink can consume? In batch, this shows up as a slow job. In streaming, it shows up as dropped messages, cascading failures, or systems that simply stop responding.",[11,964,965],{},"These aren't rare edge cases. They're Tuesday. Teams that underestimate this complexity end up with pipelines that work in demos and fail in production.",[26,967,454],{"id":453},[11,969,970],{},"Well-architected real-time systems share traits.",[11,972,973],{},"Resilience by default. Not bolted on. The system expects components to fail and continues operating. Circuit breakers. Graceful degradation. Bounded queues that shed load rather than crash.",[11,975,976],{},"Observable. You need to see what's happening inside a pipeline that processes thousands of events per second. Metrics that matter. Tracing that follows events through the system. Alerting that fires on symptoms, not just component failures.",[11,978,979],{},"Growth-ready. The system that handles ten thousand events per minute should handle ten million without a rewrite. Horizontal scaling. Partition-aware design. No single points of contention.",[11,981,982],{},"Accessible. Real-time data integration shouldn't require a PhD in distributed systems. The tools exist. The documentation is clear. The concepts are learnable. Teams should be productive in days, not quarters.",[11,984,985],{},"This last point matters more than the others. The teams that succeed with real-time infrastructure aren't the ones with the most sophisticated technology. They're the ones that made it approachable enough for their existing teams to operate.",[26,987,989],{"id":988},"the-accessibility-gap","The accessibility gap",[11,991,992],{},"There's a two-tier market forming. Tier one: companies with dedicated streaming teams, Kafka expertise, infrastructure engineers who understand partition rebalancing and exactly-once semantics. Tier two: everyone else, stuck with batch because real-time seems too complex to attempt.",[11,994,995],{},"This is backwards. Real-time data integration should be as accessible as batch processing. Same team. Same skill level. Same time-to-production. The technology is there. What's missing is the packaging. Tools that handle the complexity so teams don't have to.",[11,997,998],{},"At layline.io, we're building for the second tier. Unified workflows that handle both batch and streaming with the same interfaces. Resilience and observability built in. Scaling that happens automatically. The goal isn't to make streaming simple. It's complex, and pretending otherwise helps nobody. The goal is to make it accessible.",[11,1000,1001],{},"Because the retailers and manufacturers and financial services companies that need real-time data already have smart teams. They don't need different people. They need better tools.",[23,1003],{},[233,1005,236,1006,236,1008],{"style":235},[92,1007],{"src":239,"alt":240,"style":241},[11,1009,1010,248,1012,253],{"style":244},[246,1011,240],{},[226,1013,252],{"href":251},{"title":255,"searchDepth":256,"depth":256,"links":1015},[1016,1017,1018,1019,1020,1021],{"id":886,"depth":256,"text":887},{"id":908,"depth":256,"text":909},{"id":927,"depth":256,"text":928},{"id":949,"depth":256,"text":950},{"id":453,"depth":256,"text":454},{"id":988,"depth":256,"text":989},"2026-04-13","/images/blog/2026-04-13/real-time-data-hero.jpg",{"author":1025},{"name":240,"image":239,"url":276},"/blog/2026-04-13-why-real-time-data-integration-matters","5 min",{"title":870,"description":881},{"loc":1026},"blog/2026-04-13-why-real-time-data-integration-matters","nSvd0GKq5QfyWmddX7rLV4QutmIR5uadZ-b2z5gYq48",{"id":1033,"title":1034,"body":1035,"category":269,"date":1335,"description":1045,"extension":271,"featured":272,"image":1336,"meta":1337,"navigation":272,"path":1339,"readTime":1340,"seo":1341,"sitemap":1342,"stem":1343,"__hash__":1344},"blog/blog/2026-04-06-why-pipelines-fail-at-3am.md","Why Most Data Pipelines Fail at 3 AM (And How to Build Ones That Don't)",{"type":8,"value":1036,"toc":1320},[1037,1041,1046,1048,1052,1058,1061,1064,1068,1071,1077,1083,1089,1095,1098,1102,1105,1109,1112,1118,1124,1128,1131,1136,1141,1145,1148,1153,1158,1162,1165,1170,1175,1179,1182,1187,1192,1198,1202,1205,1208,1211,1217,1223,1229,1235,1241,1245,1248,1254,1260,1266,1272,1278,1280,1283,1286,1289,1291,1293,1296,1299,1304,1308,1310],[11,1038,1039],{},[14,1040,16],{},[11,1042,1043],{},[14,1044,1045],{},"The real reasons production data pipelines break in the middle of the night — and the engineering practices that prevent it",[23,1047],{},[26,1049,1051],{"id":1050},"the-pager-goes-off","The pager goes off",[11,1053,1054,1055],{},"It's 3:17 AM. Your phone vibrates off the nightstand. You fumble for it, squint at the brightness, and see the same message you've seen before: ",[14,1056,1057],{},"\"Data pipeline failed. Last successful run: 14 hours ago.\"",[11,1059,1060],{},"You know what happens next. You'll spend the next two hours in Slack threads, looking at logs that don't make sense, trying to figure out if this is the same failure from last week or something new. By 6 AM, you'll have a workaround running. By 9 AM, you'll tell your team it's \"handled for now.\" And by next month, you'll do it all again.",[11,1062,1063],{},"I've been there. More times than I care to admit. And after years of building data infrastructure and talking to teams who've been through the same cycle, I've noticed something: the 3 AM failures aren't random. They follow patterns. And most of them are preventable.",[26,1065,1067],{"id":1066},"why-3-am-specifically","Why 3 AM specifically?",[11,1069,1070],{},"There's nothing magical about the hour. But there is something predictable about the conditions that exist at 3 AM:",[11,1072,1073,1076],{},[246,1074,1075],{},"The human factor is at its lowest."," The engineers who built the pipeline are asleep. The operators who know the quirks are off-shift. The institutional knowledge that lives in someone's head isn't accessible. You're left with documentation that was accurate six months ago and a runbook that skips the steps that \"everyone knows.\"",[11,1078,1079,1082],{},[246,1080,1081],{},"The data volume often peaks."," Global user bases mean that \"night\" in your timezone is \"day\" somewhere else. That 3 AM failure? It's probably happening when your Asian or European users are most active. The pipeline that handled 10,000 events per minute at noon is suddenly drowning in 50,000.",[11,1084,1085,1088],{},[246,1086,1087],{},"Dependencies fail in cascading chains."," Your pipeline doesn't exist in isolation. It pulls from databases that run their own maintenance windows. It writes to APIs that have rate limits. It depends on services that deploy updates during off-peak hours. When one link breaks at 3 AM, the ripple effects hit your pipeline before anyone's awake to notice.",[11,1090,1091,1094],{},[246,1092,1093],{},"Batch jobs stack up."," That 2 AM ETL job runs fine until the day it doesn't. Maybe the source system was slower. Maybe the data volume was higher. Maybe a network hiccup added 20 minutes of latency. Suddenly your 2 AM job is still running at 3 AM, and the 3 AM job — the one your dashboard depends on — starts anyway, creating a race condition that corrupts half your data.",[11,1096,1097],{},"The 3 AM failure isn't a single bug. It's the intersection of multiple design decisions that were fine in isolation but catastrophic together.",[26,1099,1101],{"id":1100},"the-five-patterns-that-cause-most-night-failures","The five patterns that cause most night failures",[11,1103,1104],{},"After watching dozens of teams debug these incidents, I've identified five recurring patterns:",[339,1106,1108],{"id":1107},"_1-the-silent-failure","1. The silent failure",[11,1110,1111],{},"The job reports \"success\" but produced garbage data. No alerts fired because the pipeline didn't crash — it just silently did the wrong thing. You don't find out until someone in the morning asks why yesterday's revenue numbers look like a phone number.",[11,1113,1114,1117],{},[246,1115,1116],{},"Why it happens at night:"," Daytime failures get caught by humans who look at dashboards and notice anomalies. Nighttime failures wait until morning.",[11,1119,1120,1123],{},[246,1121,1122],{},"The fix:"," Validation gates. Every pipeline should have explicit data quality checks that fail the job if outputs don't meet expectations. Row counts within expected ranges. Null rates below thresholds. Referential integrity checks. If the data is wrong, the pipeline should fail loudly, not succeed quietly.",[339,1125,1127],{"id":1126},"_2-the-resource-starvation","2. The resource starvation",[11,1129,1130],{},"Your pipeline worked fine in staging. It worked fine for months in production. Then one day, the data volume hit a threshold you didn't know existed, and suddenly you're out of memory, out of disk, or out of API quota.",[11,1132,1133,1135],{},[246,1134,1116],{}," Many resource limits are soft until they're not. Memory leaks accumulate. Log files grow. Temp tables fill up. The 3 AM job is the one that finally hits the wall.",[11,1137,1138,1140],{},[246,1139,1122],{}," Resource monitoring with proactive limits. Don't just monitor whether the job finished — monitor memory usage trends, disk space trajectories, API quota consumption. Set alerts at 70% thresholds, not 100%. And design for graceful degradation: if you can't process everything, can you process the most important subset?",[339,1142,1144],{"id":1143},"_3-the-external-dependency-timeout","3. The external dependency timeout",[11,1146,1147],{},"Your pipeline calls an API. Usually it responds in 200ms. Tonight it's taking 30 seconds. Your default timeout is 60 seconds, so the job doesn't fail immediately — it just slows to a crawl. By the time it times out, it's holding locks on resources that other jobs need.",[11,1149,1150,1152],{},[246,1151,1116],{}," Third-party services do maintenance during off-peak hours. Network paths get rerouted. DNS propagates. The infrastructure you don't control changes without warning.",[11,1154,1155,1157],{},[246,1156,1122],{}," Circuit breakers and timeouts tuned to reality. If 99% of API calls complete in under 5 seconds, set your timeout to 10 seconds, not 60. Implement circuit breakers that fail fast when a dependency is struggling. Design retry logic with exponential backoff, not immediate retries that hammer a struggling service.",[339,1159,1161],{"id":1160},"_4-the-state-mismatch","4. The state mismatch",[11,1163,1164],{},"Your pipeline processes events in order. But tonight, events arrived out of order. Or duplicate events arrived. Or events arrived with timestamps that don't make sense relative to each other. Your stateful aggregation produced nonsense because the assumptions about event ordering were violated.",[11,1166,1167,1169],{},[246,1168,1116],{}," Distributed systems are eventually consistent. Network partitions happen. Message queues reorder under load. The invariants you assumed — \"events arrive in order,\" \"events arrive exactly once\" — are guarantees your infrastructure doesn't actually provide.",[11,1171,1172,1174],{},[246,1173,1122],{}," Defensive state management. Use event-time processing, not processing-time. Handle out-of-order events with watermarks. Design for at-least-once semantics and make your aggregations idempotent. Assume events will be late, duplicated, or missing — and handle it gracefully.",[339,1176,1178],{"id":1177},"_5-the-configuration-drift","5. The configuration drift",[11,1180,1181],{},"The pipeline worked yesterday. Nothing changed in the code. But someone updated an environment variable. Or rotated a credential. Or changed a database schema without updating the pipeline. The code is the same, but the world it runs in shifted.",[11,1183,1184,1186],{},[246,1185,1116],{}," Infrastructure changes often deploy during maintenance windows. Schema migrations run at off-peak hours. Credential rotations happen on schedules. The 3 AM job is the first to encounter the new world.",[11,1188,1189,1191],{},[246,1190,1122],{}," Configuration as code, tested like code. Every environment variable, every secret reference, every schema assumption should be version-controlled and validated. Run pipelines in a \"dry run\" mode after infrastructure changes. Alert on schema drift. Treat configuration changes with the same rigor as code changes.",[11,1193,1194],{},[92,1195],{"alt":1196,"src":1197},"Engineers collaborating with the five resilience patterns","/images/blog/2026-04-06/five-resilience-patterns.jpg",[26,1199,1201],{"id":1200},"the-mindset-shift-from-handle-failures-to-prevent-them","The mindset shift: from \"handle failures\" to \"prevent them\"",[11,1203,1204],{},"Most data teams I know operate in reactive mode. The pipeline fails. They fix it. They document what happened. They move on. Then it fails again, for a slightly different reason, and the cycle repeats.",[11,1206,1207],{},"The teams that don't get paged at 3 AM have a different approach. They think in terms of failure domains and blast radius. They ask: \"If this component fails, what else breaks?\" They design for graceful degradation rather than perfect reliability.",[11,1209,1210],{},"Here's what that looks like in practice:",[11,1212,1213,1216],{},[246,1214,1215],{},"Test failures, not just success paths."," Your test suite should include scenarios where dependencies time out, data is malformed, and resources are exhausted. If you only test the happy path, you're not testing production.",[11,1218,1219,1222],{},[246,1220,1221],{},"Observability over monitoring."," Monitoring tells you that a job failed. Observability tells you why. Invest in tracing that follows events through your pipeline. Log context, not just events. Build dashboards that show the health of data quality, not just job completion.",[11,1224,1225,1228],{},[246,1226,1227],{},"Chaos engineering for data pipelines."," If you haven't deliberately broken your pipeline in a controlled way, you don't know how it fails. Run drills where you kill database connections, introduce latency, and corrupt input data. Learn your failure modes before they learn you.",[11,1230,1231,1234],{},[246,1232,1233],{},"On-call that escalates to people who can fix it."," The person who gets paged at 3 AM should be someone who can actually fix the problem, not just restart the job and hope. If your on-call rotation is too junior, you're just delaying the real fix until morning anyway.",[11,1236,1237],{},[92,1238],{"alt":1239,"src":1240},"Well-designed systems let engineers sleep peacefully","/images/blog/2026-04-06/sleeping-peacefully.jpg",[26,1242,1244],{"id":1243},"building-pipelines-that-sleep-through-the-night","Building pipelines that sleep through the night",[11,1246,1247],{},"Resilient data pipelines share common traits. They're not magic — they're engineered with specific patterns:",[11,1249,1250,1253],{},[246,1251,1252],{},"Idempotency everywhere."," Running the same job twice should produce the same result as running it once. This makes retries safe and recovery automatic.",[11,1255,1256,1259],{},[246,1257,1258],{},"Backpressure handling."," When downstream systems can't keep up, the pipeline should slow down, not crash or drop data. It should shed load gracefully, not catastrophically.",[11,1261,1262,1265],{},[246,1263,1264],{},"Bounded state."," Stateful operations should have limits. Windowed aggregations with TTL. State stores with eviction policies. Don't let unbounded state become an unbounded problem.",[11,1267,1268,1271],{},[246,1269,1270],{},"Explicit contracts."," Define and validate schemas at pipeline boundaries. Reject malformed data early. Fail fast when assumptions are violated.",[11,1273,1274,1277],{},[246,1275,1276],{},"Operational runbooks that work."," Every alert should have a runbook. Every runbook should be tested. If the runbook says \"check the logs,\" specify which logs, what to look for, and what to do when you find it.",[26,1279,198],{"id":197},[11,1281,1282],{},"The 3 AM pager doesn't have to be inevitable. It's a symptom of design choices that prioritized throughput over resilience, completion over correctness, and feature velocity over operational maturity.",[11,1284,1285],{},"The teams that sleep through the night aren't luckier. They've invested in the unglamorous work of error handling, validation, and observability. They've accepted that failures will happen and designed systems that handle them gracefully.",[11,1287,1288],{},"Your users don't care if your pipeline was clever. They care if the data is right when they need it. Build for that.",[23,1290],{},[26,1292,213],{"id":212},[11,1294,1295],{},"If you're tired of 3 AM pages, start with one change: add a single validation gate to your most critical pipeline. Check row counts. Verify null rates. Validate a key business metric. Make the pipeline fail if the data looks wrong.",[11,1297,1298],{},"It's not a complete solution, but it's a start. And once you've felt the relief of catching a data quality issue before it reaches your users, you'll be motivated to add the next safeguard.",[11,1300,1301],{},[14,1302,1303],{},"For teams building streaming pipelines, layline.io provides built-in backpressure handling, exactly-once semantics, and visual debugging that makes it easier to understand what's happening when things go wrong — whether that's at 3 PM or 3 AM. The Community Edition is free to explore.",[11,1305,1306],{},[226,1307,229],{"href":228},[23,1309],{},[233,1311,236,1312,236,1314],{"style":235},[92,1313],{"src":239,"alt":240,"style":241},[11,1315,1316,248,1318,253],{"style":244},[246,1317,240],{},[226,1319,252],{"href":251},{"title":255,"searchDepth":256,"depth":256,"links":1321},[1322,1323,1324,1331,1332,1333,1334],{"id":1050,"depth":256,"text":1051},{"id":1066,"depth":256,"text":1067},{"id":1100,"depth":256,"text":1101,"children":1325},[1326,1327,1328,1329,1330],{"id":1107,"depth":540,"text":1108},{"id":1126,"depth":540,"text":1127},{"id":1143,"depth":540,"text":1144},{"id":1160,"depth":540,"text":1161},{"id":1177,"depth":540,"text":1178},{"id":1200,"depth":256,"text":1201},{"id":1243,"depth":256,"text":1244},{"id":197,"depth":256,"text":198},{"id":212,"depth":256,"text":213},"2026-04-06","/images/blog/2026-04-06/3am-pipeline-failure-hero.jpg",{"author":1338},{"name":240,"image":239,"url":276},"/blog/2026-04-06-why-pipelines-fail-at-3am","7 min",{"title":1034,"description":1045},{"loc":1339},"blog/2026-04-06-why-pipelines-fail-at-3am","tYvwq8p5le3WH2Br-zsKPorQ6ATHusBrO1vXM9SJZ7c",{"id":1346,"title":1347,"body":1348,"category":269,"date":1590,"description":1591,"extension":271,"featured":272,"image":1592,"meta":1593,"navigation":272,"path":1604,"readTime":1605,"seo":1606,"sitemap":1607,"stem":1608,"__hash__":1609},"blog/blog/2026-03-30-from-batch-regret-to-real-time-decision-framework.md","The Streaming Migration Nobody Asked For",{"type":8,"value":1349,"toc":1582},[1350,1354,1357,1360,1363,1366,1370,1373,1376,1382,1385,1388,1392,1395,1401,1407,1413,1419,1422,1426,1429,1435,1441,1447,1453,1456,1460,1463,1469,1475,1481,1487,1491,1494,1500,1506,1516,1526,1529,1533,1536,1539,1542,1549,1552,1555,1557,1570,1572],[11,1351,1352],{},[14,1353,16],{},[11,1355,1356],{},"I was at a conference last year when a data architect — Fortune 500, big team, serious budget — pulled me aside and told me a story I've heard too many times now.",[11,1358,1359],{},"They spent 14 months migrating to Kafka. Hired a new team. Stood up new infrastructure. Built a whole new on-call rotation. The works.",[11,1361,1362],{},"Six months after launch, they quietly moved half the pipelines back to batch.",[11,1364,1365],{},"Kafka didn't break. It worked fine. The problem was dumber than that: nobody used the real-time data. The dashboards still got checked at 9 AM. Reports still ran weekly. The ML models still retrained overnight. They'd built a Formula 1 car to drive to the grocery store.",[26,1367,1369],{"id":1368},"the-conference-talk-that-launched-a-thousand-migrations","The Conference Talk That Launched a Thousand Migrations",[11,1371,1372],{},"Here's how it usually starts. Someone on the team watches a conference talk — probably at 2x speed on YouTube — where a FAANG engineer describes their real-time pipeline. Billions of events per second. Sub-millisecond latency. Dashboards updating like the Matrix.",[11,1374,1375],{},"That engineer comes back to the office inspired. \"We should do this.\" The team nods. The CTO loves the word \"real-time\" in the quarterly roadmap. A Jira epic is born.",[11,1377,1378,1379],{},"What nobody asks: ",[14,1380,1381],{},"does anyone in this building actually need data faster than they're getting it today?",[11,1383,1384],{},"Not \"would it be nice.\" Not \"it sounds more modern.\" Does a specific person, making a specific decision, get measurably worse outcomes because the data is an hour old instead of a second old?",[11,1386,1387],{},"Nine times out of ten, the honest answer is no. And the one time it's yes, it's usually one pipeline — not the whole platform.",[26,1389,1391],{"id":1390},"the-grocery-list","The Grocery List",[11,1393,1394],{},"Before you migrate anything, make a list. I'm serious — open a spreadsheet. For each pipeline, answer three questions:",[11,1396,1397],{},[92,1398],{"alt":1399,"src":1400},"A team of engineers working through a pipeline checklist together","/images/blog/2026-03-30/pipeline-checklist-team.jpg",[11,1402,1403,1406],{},[246,1404,1405],{},"Who consumes this data?"," A name. A team. A system. If you can't name the consumer, the pipeline might not need to exist at all, let alone in real-time.",[11,1408,1409,1412],{},[246,1410,1411],{},"What do they do with it, and when?"," If the answer is \"they check a dashboard every morning\" or \"it feeds a report on Fridays,\" streaming won't change the outcome. You're just making the plumbing more expensive for the same water.",[11,1414,1415,1418],{},[246,1416,1417],{},"What breaks if this data is 1 hour late? 1 day late?"," This is the only question that actually separates batch from streaming. If the answer to both is \"nothing, really,\" you've got a batch workload wearing a streaming costume.",[11,1420,1421],{},"Most teams discover that 80% of their pipelines are perfectly fine as batch. The remaining 20% is where things get interesting.",[26,1423,1425],{"id":1424},"where-speed-actually-matters","Where Speed Actually Matters",[11,1427,1428],{},"Some data genuinely spoils. Like milk, not wine.",[11,1430,1431,1434],{},[246,1432,1433],{},"Fraud detection"," is the obvious one. A credit card transaction flagged five minutes after the fact isn't fraud detection — it's fraud notification. The money's already gone. If your fraud pipeline runs in batch, you're writing apology letters instead of blocking doors.",[11,1436,1437,1440],{},[246,1438,1439],{},"Operational alerts"," — if your IoT sensor tells you a turbine is overheating, that information has a shelf life measured in seconds. An hourly batch job here isn't just slow. It's negligent.",[11,1442,1443,1446],{},[246,1444,1445],{},"Pricing and inventory"," in competitive markets. If your competitor updates prices every 30 seconds and you update every 6 hours, you're not competing. You're spectating.",[11,1448,1449,1452],{},[246,1450,1451],{},"Multi-consumer event streams"," where the economics compound. One Kafka topic feeding three downstream systems can be cheaper than three separate batch jobs pulling from the same database. Streaming earns its keep here not through speed, but through architecture elegance.",[11,1454,1455],{},"The pattern: in every case, there's a specific, measurable cost to delay. Not a vague feeling. A number.",[26,1457,1459],{"id":1458},"the-costs-nobody-puts-in-the-jira-epic","The Costs Nobody Puts in the Jira Epic",[11,1461,1462],{},"Streaming infrastructure has a pricing model that looks great on the vendor slide and terrible on the Q3 actuals.",[11,1464,1465,1468],{},[246,1466,1467],{},"It runs 24/7."," Your batch job runs for 4 hours and sleeps. Your streaming job runs all day, all night, weekends, holidays. Even if the total data volume is identical, the compute bill isn't. A team I know went from a $2,000/month batch setup to $11,000/month streaming — processing the same data, delivering it to the same place, consumed at the same cadence.",[11,1470,1471,1474],{},[246,1472,1473],{},"Debugging goes from archaeology to quantum physics."," When a batch job fails, you get a stack trace, a bad record, and a clear rerun path. When a streaming job produces wrong output, it might not \"fail\" at all. It just silently feeds garbage downstream until someone notices the revenue dashboard looks weird three days later.",[11,1476,1477,1480],{},[246,1478,1479],{},"Schema changes become a diplomatic negotiation."," In batch, you version your code, test on historical data, and push. In streaming, changing a field means coordinating every producer and consumer simultaneously on a live system. Get the ordering wrong and you're debugging data corruption at 2 AM.",[11,1482,1483,1486],{},[246,1484,1485],{},"The on-call tax is real."," Consumer lag, partition skew, broker failovers — these aren't rare edge cases, they're Tuesday. If your team is already stretched thin, streaming doesn't solve the problem. It multiplies it.",[26,1488,1490],{"id":1489},"a-framework-that-actually-helps","A Framework That Actually Helps",[11,1492,1493],{},"Here's what I'd recommend. It takes about two hours with the right people in the room.",[11,1495,1496,1499],{},[246,1497,1498],{},"Step 1: Name the decisions."," For each pipeline, identify the specific business decision it supports. Not \"analytics\" — the actual decision. \"Approve or decline this transaction.\" \"Reorder this SKU.\" \"Alert the maintenance crew.\" If you can't name it, batch is fine.",[11,1501,1502,1505],{},[246,1503,1504],{},"Step 2: Time the decisions."," How often does that decision get made? Every second? Every hour? Every Monday? Match the pipeline cadence to the decision cadence. Sub-second delivery for a daily decision is waste.",[11,1507,1508,1511,1512,1515],{},[246,1509,1510],{},"Step 3: Price the delay."," What does a one-hour delay cost, in dollars? This is the hardest and most important question. If the answer is \"we don't know\" or \"probably nothing,\" you don't have a streaming use case. You have a streaming ",[14,1513,1514],{},"wish",".",[11,1517,1518,1521,1522,1525],{},[246,1519,1520],{},"Step 4: Start with one."," Pick the pipeline with the clearest delay cost. Migrate it. Run it alongside the batch version for a full cycle. Compare. Fix what breaks. ",[14,1523,1524],{},"Then"," decide whether to expand.",[11,1527,1528],{},"Teams that do this well usually end up with 2–3 streaming pipelines and 15 batch ones. Teams that skip this process end up with 18 streaming pipelines, a burned-out ops team, and a quiet migration back to batch six months later.",[26,1530,1532],{"id":1531},"its-not-eitheror","It's Not Either/Or",[11,1534,1535],{},"Here's the thing most streaming vendors won't tell you: the best data architectures are boring. They're not all-streaming or all-batch. They're a mix, chosen pipeline by pipeline, based on what the data actually needs to do.",[11,1537,1538],{},"Batch for reporting. Batch for ML training. Batch for anything where \"daily\" is fast enough and simplicity saves you $8,000 a month in ops overhead.",[11,1540,1541],{},"Streaming for fraud. Streaming for operational alerts. Streaming for the few pipelines where delay has a dollar sign attached.",[11,1543,1544,1545,1548],{},"This is exactly why we built ",[226,1546,252],{"href":1547},"/"," the way we did. It handles both batch and streaming in the same platform — same workflows, same tooling, same team. You don't have to pick one world and abandon the other. You start with what you have, add real-time where it earns its keep, and keep batch where it makes sense. No rip-and-replace. No either/or.",[11,1550,1551],{},"The architecture diagram won't win any conference talks. But the team goes home at 5 PM and the on-call rotation actually sleeps through the night.",[11,1553,1554],{},"That's the kind of boring that scales.",[23,1556],{},[11,1558,1559],{},[14,1560,1561,1562,1564,1565,1569],{},"If you're evaluating which pipelines belong in real-time and which are perfectly fine as batch, ",[226,1563,252],{"href":1547}," lets you run both in a single platform — so you can validate the business case before committing infrastructure. ",[226,1566,1568],{"href":1567},"/resources/contact","Talk to us"," about your specific architecture.",[23,1571],{},[233,1573,236,1574,236,1576],{"style":235},[92,1575],{"src":239,"alt":240,"style":241},[11,1577,1578,248,1580,253],{"style":244},[246,1579,240],{},[226,1581,252],{"href":251},{"title":255,"searchDepth":256,"depth":256,"links":1583},[1584,1585,1586,1587,1588,1589],{"id":1368,"depth":256,"text":1369},{"id":1390,"depth":256,"text":1391},{"id":1424,"depth":256,"text":1425},{"id":1458,"depth":256,"text":1459},{"id":1489,"depth":256,"text":1490},{"id":1531,"depth":256,"text":1532},"2026-03-30","A data architect at a Fortune 500 once told me they spent 14 months migrating to Kafka — then quietly switched half the pipelines back to batch. Here's why that keeps happening, and what I'd do instead.","/images/blog/2026-03-30/streaming-migration-hero.jpg",{"tags":1594,"keywords":1599,"author":1603},[1595,1596,1597,1598],"data architecture","real-time streaming","decision framework","batch processing",[1600,1601,1602],"batch to real-time data architecture decision","when to choose streaming over batch processing","real-time vs batch processing tradeoffs",{"name":240,"image":239,"url":276},"/blog/2026-03-30-from-batch-regret-to-real-time-decision-framework","6 min",{"title":1347,"description":1591},{"loc":1604},"blog/2026-03-30-from-batch-regret-to-real-time-decision-framework","w8SKJzB6ybAvIv86Rd4ymbPUgkoW7Q1RlZbcMJd0Vko",{"id":1611,"title":1612,"body":1613,"category":269,"date":1875,"description":1876,"extension":271,"featured":272,"image":1877,"meta":1878,"navigation":272,"path":1879,"readTime":278,"seo":1880,"sitemap":1881,"stem":1882,"__hash__":1883},"blog/blog/2026-03-23-from-airflow-to-real-time-migration-guide.md","When Your Airflow Pipelines Need to Go Real-Time",{"type":8,"value":1614,"toc":1864},[1615,1620,1622,1626,1640,1643,1646,1650,1653,1659,1665,1671,1674,1678,1681,1687,1693,1699,1705,1708,1712,1715,1721,1727,1733,1739,1743,1746,1749,1755,1761,1767,1773,1779,1782,1788,1792,1795,1798,1801,1804,1808,1811,1814,1817,1820,1824,1827,1830,1833,1836,1839,1843,1846,1849,1852,1855,1860],[11,1616,1617],{},[14,1618,1619],{},"You've got Airflow running. Your team knows it. The DAGs work. So why are you suddenly hearing \"we need real-time\" — and what do you actually do about it?",[23,1621],{},[26,1623,1625],{"id":1624},"the-ask-that-wont-go-away","The ask that won't go away",[11,1627,1628,1629,1632,1633,1636,1637],{},"It comes from the business side first. Usually via Slack: ",[14,1630,1631],{},"\"Can we get that dashboard updated more than once a day?\""," Then from product: ",[14,1634,1635],{},"\"The fraud team wants to know about issues within seconds, not hours.\""," Then from your CTO in the next planning meeting: ",[14,1638,1639],{},"\"Why are we still running batch when our competitors are doing real-time?\"",[11,1641,1642],{},"You're the Airflow person. You've built a solid batch operation. Your DAGs run on schedule. Your team can debug them. You've got the runbooks. And now everyone's asking you to become a streaming engineer overnight.",[11,1644,1645],{},"This guide is for that moment. Not the pitch for why real-time matters — you've probably already accepted that. This is about what you actually do with your existing Airflow setup when the real-time ask arrives.",[26,1647,1649],{"id":1648},"where-airflow-hits-its-ceiling","Where Airflow hits its ceiling",[11,1651,1652],{},"Airflow is a workflow orchestrator. It runs tasks on schedules or triggers. That's extremely useful — and the right tool for a lot of them. But there are genuine use cases where it starts to show limits.",[11,1654,1655,1658],{},[246,1656,1657],{},"Latency is the obvious one."," If your shortest schedule is 15 minutes, everything downstream waits 15 minutes minimum. For some workflows — daily reports, bulk API syncs, ML training pipelines — that's perfectly fine. For others — fraud alerts, inventory updates, user notifications — 15 minutes is an eternity.",[11,1660,1661,1664],{},[246,1662,1663],{},"High-frequency triggers get expensive."," Scheduling a task every few seconds falls apart when you need to react to thousands of events per second. You end up with sensor tasks polling for conditions, which is not what Airflow was designed for.",[11,1666,1667,1670],{},[246,1668,1669],{},"State between events is awkward."," Airflow tasks are stateless and short-lived. If you need to maintain state across millions of individual events — tracking session windows, building real-time aggregations, handling out-of-order arrivals — you're fighting the paradigm.",[11,1672,1673],{},"None of this is a criticism of Airflow. It's about knowing when to reach for a different tool.",[26,1675,1677],{"id":1676},"the-four-realistic-options","The four realistic options",[11,1679,1680],{},"When teams ask \"should we move to streaming?\", the real question is usually \"what's the lowest-cost path to real-time capability?\" Here are the four paths teams actually take:",[11,1682,1683,1686],{},[246,1684,1685],{},"1. Keep everything in Airflow, but schedule more frequently.","\nFor some teams, running DAGs every 5 minutes is enough. If \"real-time\" means \"within a few minutes,\" cron-level scheduling can get you there without adding any new infrastructure. Don't assume you need Kafka to react faster — check if your current tool can already do it.",[11,1688,1689,1692],{},[246,1690,1691],{},"2. Add a streaming layer alongside Airflow.","\nThis is the most common path for mature teams. You keep Airflow for batch workflows, complex dependency trees, and anything with human-in-the-loop steps. You add a streaming platform for event-driven, low-latency workloads. They coexist.",[11,1694,1695,1698],{},[246,1696,1697],{},"3. Migrate specific pipelines entirely to streaming.","\nSometimes a workflow that runs in Airflow shouldn't be in Airflow at all — it was just the only tool available when it was built. For high-volume, event-driven pipelines, a full migration to streaming infrastructure makes sense.",[11,1700,1701,1704],{},[246,1702,1703],{},"4. Replace Airflow entirely.","\nRarely the right call, but it happens when an organization is committing fully to an event-driven architecture and wants one system handling everything. The migration cost is high and the risk is real.",[11,1706,1707],{},"Most teams end up doing option 2 or 3 for specific pipelines. That's the practical reality.",[26,1709,1711],{"id":1710},"how-to-assess-what-you-actually-have","How to assess what you actually have",[11,1713,1714],{},"Before you plan anything, map the actual workload. Not in theory — in practice.",[11,1716,1717,1720],{},[246,1718,1719],{},"Find the latency-sensitive pipelines."," Which DAGs feed downstream systems that customers or users directly interact with? Which ones serve data that changes business outcomes if it's 5 minutes old versus 5 seconds? Start there.",[11,1722,1723,1726],{},[246,1724,1725],{},"Count the handoffs."," Look at pipelines where data moves through multiple DAGs in sequence. Each handoff adds latency and failure surface. Streaming can often collapse multiple batch steps into one continuous flow.",[11,1728,1729,1732],{},[246,1730,1731],{},"Talk to the consumers."," Not the engineering leads — the actual business users. Ask them what \"real-time\" means to them. You'll often find that \"real-time\" to the business is \"within an hour\" to them, and that changes the priority entirely.",[11,1734,1735,1738],{},[246,1736,1737],{},"Assess your operational capacity."," Streaming introduces different failure modes: consumer lag, partition skew, broker disk usage. If your team is already at capacity maintaining batch pipelines, adding streaming without headroom will create problems.",[26,1740,1742],{"id":1741},"a-migration-that-doesnt-break-everything","A migration that doesn't break everything",[11,1744,1745],{},"The worst way to migrate is to treat it as a rewrite. Rip out the DAG, build the streaming version, hope it works in production.",[11,1747,1748],{},"The practical path is incremental.",[11,1750,1751,1754],{},[246,1752,1753],{},"Step 1: Shadow mode.","\nPick one batch pipeline with real latency sensitivity. Build the streaming version alongside it. Route the streaming output to a test or staging consumer, not production. Let them run in parallel for at least one full cycle.",[11,1756,1757,1760],{},[246,1758,1759],{},"Step 2: Validate.","\nDoes the streaming pipeline produce the same results as the batch version? For aggregations, this means comparing numbers. For event routing, this means verifying that every expected event reached the expected destination. Don't skip this step.",[11,1762,1763,1766],{},[246,1764,1765],{},"Step 3: Dual-write period.","\nPoint a non-critical production consumer at the streaming output while keeping the batch output as the primary source. Monitor error rates, latency distributions, and consumer lag. Fix what breaks.",[11,1768,1769,1772],{},[246,1770,1771],{},"Step 4: Switch over.","\nAfter a successful dual-write period, make the streaming output primary. Keep the batch pipeline running in standby for a defined period — a week, two weeks — before decommissioning.",[11,1774,1775,1778],{},[246,1776,1777],{},"Step 5: Repeat.","\nApply the lessons. Each migration is faster than the last.",[11,1780,1781],{},"The hybrid period isn't optional — it's how you maintain confidence in the data while you're validating the new system.",[11,1783,1784],{},[92,1785],{"alt":1786,"src":1787},"Side-by-side: batch and streaming running together during migration","/images/blog/2026-03-23/airflow-to-streaming-migration-path.jpg",[26,1789,1791],{"id":1790},"what-about-the-airflow-dags-youve-already-built","What about the Airflow DAGs you've already built?",[11,1793,1794],{},"This is the question nobody answers well. The reality: your existing DAGs represent accumulated knowledge about your data and workflows. Don't throw that away.",[11,1796,1797],{},"Some DAGs should migrate to streaming. Others should stay batch — because the workflow is genuinely batch-oriented, the latency requirement is real but manageable at hourly intervals, or the transformation logic is complex enough that rebuilding it isn't worth the engineering cost.",[11,1799,1800],{},"A useful heuristic: if the pipeline exists primarily to move data from A to B on a schedule, it might be a streaming candidate. If it exists to orchestrate multi-step transformations with conditional logic and human approval gates, Airflow is probably still the right home.",[11,1802,1803],{},"The goal isn't to replace Airflow. It's to add streaming where it earns its keep — and let each tool do what it's actually good at.",[26,1805,1807],{"id":1806},"what-good-looks-like-when-its-working","What good looks like when it's working",[11,1809,1810],{},"When the migration works, the business notices — not the infrastructure.",[11,1812,1813],{},"A fraud analyst who used to review flagged transactions six hours after they occurred is now reviewing them in under a minute. A product manager who checked the dashboard each morning for yesterday's numbers is now seeing updates as events happen. These are the outcomes worth optimizing for.",[11,1815,1816],{},"The infrastructure teams notice too, but in a different way: fewer emergency pages about batch job failures, more time on improvement work, observability dashboards that show exactly where data is flowing and where it's backing up.",[11,1818,1819],{},"Faster decisions. Less manual babysitting. More time building things that matter",[26,1821,1823],{"id":1822},"before-you-start","Before you start",[11,1825,1826],{},"Get clear on a few things before you write the first line of streaming logic:",[11,1828,1829],{},"What's the actual cost of latency in your most important workflow? Not an assumption — actual numbers. If the fraud pipeline takes 6 hours instead of 6 seconds, what's the financial impact? That's your prioritization signal.",[11,1831,1832],{},"What's your rollback plan? If the streaming pipeline breaks at 2 AM, what happens? Automatic fallback to batch? Manual intervention? PagerDuty escalation? Define this before you launch, not after.",[11,1834,1835],{},"What's the team's learning curve? You'll need to learn new concepts: consumer groups, partition keys, offset management, watermark policies. Make sure your team has time allocated to understand these — not just implement them.",[11,1837,1838],{},"And if the honest answer is that your team doesn't have the bandwidth to operate a streaming system alongside existing Airflow pipelines right now — that's fine. Say so. Streaming urgency from the business is often lower than the business thinks, and overcommitting your team to a migration you can't support is worse than saying no.",[26,1840,1842],{"id":1841},"the-practical-path-forward","The practical path forward",[11,1844,1845],{},"The teams that do this well share one trait: they don't try to boil the ocean.",[11,1847,1848],{},"They pick one high-value, latency-sensitive pipeline. They build it in streaming alongside the existing batch version. They validate rigorously. They cut over when they're confident. Then they do the next one.",[11,1850,1851],{},"Airflow stays. It handles what it's good at. Streaming gets added where the latency value is real and measurable. The result is an architecture that uses the right tool for each workload — not a big-bang migration that bets everything on a single weekend rewrite.",[11,1853,1854],{},"Start with one pipeline. Get it right. Learn what you don't know. Then scale from there.",[11,1856,1857],{},[14,1858,1859],{},"If you're evaluating platforms for the streaming layer, layline.io offers a visual workflow designer that lets you prototype and deploy streaming pipelines without requiring distributed systems expertise. The Community Edition is free to try — no credit card required.",[11,1861,1862],{},[226,1863,229],{"href":228},{"title":255,"searchDepth":256,"depth":256,"links":1865},[1866,1867,1868,1869,1870,1871,1872,1873,1874],{"id":1624,"depth":256,"text":1625},{"id":1648,"depth":256,"text":1649},{"id":1676,"depth":256,"text":1677},{"id":1710,"depth":256,"text":1711},{"id":1741,"depth":256,"text":1742},{"id":1790,"depth":256,"text":1791},{"id":1806,"depth":256,"text":1807},{"id":1822,"depth":256,"text":1823},{"id":1841,"depth":256,"text":1842},"2026-03-23","You've got Airflow running. Your team knows it. The DAGs work. So why are you suddenly hearing 'we need real-time' — and what do you actually do about it?","/images/blog/2026-03-23/airflow-to-streaming-hero.jpg",{},"/blog/2026-03-23-from-airflow-to-real-time-migration-guide",{"title":1612,"description":1876},{"loc":1879},"blog/2026-03-23-from-airflow-to-real-time-migration-guide","dk6_Qo1_jdR7JZiVlLuOQ-ry0oczyniRdGBO8h1s5fg",{"id":1885,"title":1886,"body":1887,"category":269,"date":2198,"description":2199,"extension":271,"featured":272,"image":2200,"meta":2201,"navigation":272,"path":2202,"readTime":553,"seo":2203,"sitemap":2204,"stem":2205,"__hash__":2206},"blog/blog/2026-03-18-edge-processing-saves-money.md","Edge Processing That Actually Saves Money",{"type":8,"value":1888,"toc":2190},[1889,1892,1895,1898,1902,1905,1908,1934,1941,1944,1948,1951,1954,1957,2028,2034,2037,2041,2044,2050,2055,2066,2071,2088,2091,2095,2098,2124,2127,2138,2142,2145,2165,2169,2172,2175,2177],[11,1890,1891],{},"The invoice arrived like it did every month. But this time, the number stopped them cold: $47,000 in cloud egress fees for moving unprocessed IoT data from the factory floor to the data lake.",[11,1893,1894],{},"Across industries, teams building real-time data pipelines discover a brutal truth: sending raw data to the cloud is expensive. Really expensive. And it's getting worse as sensor volumes explode.",[11,1896,1897],{},"Here's the math most people miss — and the solution that changes how teams think about data movement.",[26,1899,1901],{"id":1900},"the-problem-no-one-talks-about","The Problem No One Talks About",[11,1903,1904],{},"Cloud egress fees are the silent killer of data initiatives. When you're streaming millions of sensor readings per minute from industrial equipment, vehicles, or retail locations, you're not just paying for compute. You're paying to move all that data out of the cloud provider's network.",[11,1906,1907],{},"For a mid-sized industrial deployment:",[359,1909,1910,1916,1922,1928],{},[362,1911,1912,1915],{},[246,1913,1914],{},"10,000 sensors"," generating readings every 30 seconds",[362,1917,1918,1921],{},[246,1919,1920],{},"Average payload",": 500 bytes per reading",[362,1923,1924,1927],{},[246,1925,1926],{},"Monthly data volume",": ~1.3 GB/month per sensor",[362,1929,1930,1933],{},[246,1931,1932],{},"At $0.09/GB egress",": that's $117/month per sensor",[11,1935,1936,1937,1940],{},"Scale to 10,000 sensors, and you're looking at ",[246,1938,1939],{},"$1.17 million annually",". Just$$ for moving bytes.",[11,1942,1943],{},"The kicker? Most of that data isn't useful in its raw form. It's noise — temperature fluctuations within normal ranges, redundant GPS pings, sensor drift that needs filtering before it tells you anything meaningful.",[26,1945,1947],{"id":1946},"what-edge-processing-actually-means","What Edge Processing Actually Means",[11,1949,1950],{},"Edge processing isn't a buzzword. It's running your data logic closer to where the data is born — on-premises, on the device, or at the network edge — before anything hits the cloud.",[11,1952,1953],{},"Instead of sending every single sensor reading upstream, you filter, aggregate, and transform at the edge. Only relevant insights or aggregated metrics travel across the wire.",[11,1955,1956],{},"The math shifts dramatically:",[1958,1959,1960,1976],"table",{},[1961,1962,1963],"thead",{},[1964,1965,1966,1970,1973],"tr",{},[1967,1968,1969],"th",{},"Metric",[1967,1971,1972],{},"Cloud-Only",[1967,1974,1975],{},"Edge + Cloud",[1977,1978,1979,1991,2002,2013],"tbody",{},[1964,1980,1981,1985,1988],{},[1982,1983,1984],"td",{},"Raw data sent",[1982,1986,1987],{},"1.3 GB/sensor/mo",[1982,1989,1990],{},"0.01 GB/sensor/mo",[1964,1992,1993,1996,1999],{},[1982,1994,1995],{},"Egress cost",[1982,1997,1998],{},"$117/sensor/mo",[1982,2000,2001],{},"$1.17/sensor/mo",[1964,2003,2004,2007,2010],{},[1982,2005,2006],{},"Annual (10K sensors)",[1982,2008,2009],{},"$1.17M",[1982,2011,2012],{},"$11,700",[1964,2014,2015,2020,2023],{},[1982,2016,2017],{},[246,2018,2019],{},"Savings",[1982,2021,2022],{},"—",[1982,2024,2025],{},[246,2026,2027],{},"99%",[11,2029,2030],{},[92,2031],{"alt":2032,"src":2033},"Edge vs Cloud Processing Comparison","/images/blog/2026-03-18/edge-processing-comparison.jpg",[11,2035,2036],{},"That's not a typo. Local filtering and aggregation can cut your bandwidth costs by 90% or more.",[26,2038,2040],{"id":2039},"a-real-example-manufacturing-floor","A Real Example: Manufacturing Floor",[11,2042,2043],{},"Consider a concrete scenario from a manufacturing deployment:",[11,2045,2046,2049],{},[246,2047,2048],{},"The setup:"," 500 machines, each with 20 sensors reporting every second. Raw data stream: 360 million records per day.",[11,2051,2052],{},[246,2053,2054],{},"Without edge processing:",[359,2056,2057,2060,2063],{},[362,2058,2059],{},"Daily cloud egress: ~180 GB",[362,2061,2062],{},"Monthly cost: ~$15,000 just in bandwidth",[362,2064,2065],{},"Plus compute costs to process all that noise",[11,2067,2068],{},[246,2069,2070],{},"With edge processing (deployed at the edge):",[359,2072,2073,2076,2079,2082],{},[362,2074,2075],{},"Each edge node filters: removes readings within normal ranges, aggregates 1-second data into 5-minute summaries",[362,2077,2078],{},"Daily cloud egress: ~4 GB",[362,2080,2081],{},"Monthly cost: ~$350",[362,2083,2084,2085],{},"Total savings: $14,650/month, or ",[246,2086,2087],{},"$175,800/year",[11,2089,2090],{},"And the processing logic at the edge is doing more than filtering — it's enriching data with local context, handling protocol conversions (OPC-UA to JSON, for example), and routing only actionable events upstream.",[26,2092,2094],{"id":2093},"when-edge-makes-sense-and-when-it-doesnt","When Edge Makes Sense (And When It Doesn't)",[11,2096,2097],{},"Edge processing isn't universal. It shines when:",[359,2099,2100,2106,2112,2118],{},[362,2101,2102,2105],{},[246,2103,2104],{},"Data volumes are massive"," — millions of events per day",[362,2107,2108,2111],{},[246,2109,2110],{},"Latency matters"," — you need sub-second responses",[362,2113,2114,2117],{},[246,2115,2116],{},"Bandwidth costs are painful"," — egress fees are a line item you want to shrink",[362,2119,2120,2123],{},[246,2121,2122],{},"Connectivity is unreliable"," — edge nodes can buffer during outages",[11,2125,2126],{},"You might skip it if:",[359,2128,2129,2132,2135],{},[362,2130,2131],{},"Data volumes are manageable (under 10 GB/day)",[362,2133,2134],{},"All processing happens in a single cloud region anyway",[362,2136,2137],{},"Your team has no on-prem or edge infrastructure capacity",[26,2139,2141],{"id":2140},"the-bigger-picture","The Bigger Picture",[11,2143,2144],{},"Cutting egress costs is the visible win. But the ripple effects matter too:",[359,2146,2147,2153,2159],{},[362,2148,2149,2152],{},[246,2150,2151],{},"Faster insights",": Processing at the edge reduces round-trip latency from seconds to milliseconds",[362,2154,2155,2158],{},[246,2156,2157],{},"Better reliability",": Local processing keeps working when connectivity drops",[362,2160,2161,2164],{},[246,2162,2163],{},"Compliance wins",": Sensitive data stays on-prem, only anonymized insights go to the cloud",[26,2166,2168],{"id":2167},"your-turn","Your Turn",[11,2170,2171],{},"If you're spending more than $5,000/month on cloud egress for streaming data, edge processing probably makes sense. Run the numbers on your own setup — filter for the data that actually matters, aggregate what you can, and send only the rest.",[11,2173,2174],{},"For teams ready to make the shift, modern edge orchestration platforms such as layline.io can handle edge deployment as a first-class concern: container-native, runs anywhere (industrial PC, gateway, Kubernetes cluster), and provides the same visual workflow for edge and cloud processing.",[23,2176],{},[11,2178,2179,2182,2183,2189],{},[246,2180,2181],{},"CTA:"," Want help modeling edge savings? Run a quick audit of current data volumes and egress costs, then compare it against what edge processing could look like. ",[226,2184,2188],{"href":2185,"rel":2186},"https://layline.io/resources/booking",[2187],"nofollow","Book a technical chat"," — a solutions engineer can walk through the numbers.",{"title":255,"searchDepth":256,"depth":256,"links":2191},[2192,2193,2194,2195,2196,2197],{"id":1900,"depth":256,"text":1901},{"id":1946,"depth":256,"text":1947},{"id":2039,"depth":256,"text":2040},{"id":2093,"depth":256,"text":2094},{"id":2140,"depth":256,"text":2141},{"id":2167,"depth":256,"text":2168},"2026-03-18","How Local Data Filtering Can Cut Your Cloud Bill by 90% — And What Yours Could Look Like","/images/blog/2026-03-18/edge-processing-hero.jpg",{},"/blog/2026-03-18-edge-processing-saves-money",{"title":1886,"description":2199},{"loc":2202},"blog/2026-03-18-edge-processing-saves-money","Yp6jtroDX5VZgIEB7Eb_IQTpgAQmr7BNKgz8fHq8ikc",{"id":2208,"title":2209,"body":2210,"category":269,"date":2579,"description":2216,"extension":271,"featured":272,"image":2580,"meta":2581,"navigation":272,"path":2582,"readTime":553,"seo":2583,"sitemap":2584,"stem":2585,"__hash__":2586},"blog/blog/2026-03-17-etl-to-streaming-migration-playbook.md","From Batch to Streaming: A Practical Guide to Modern Data Pipelines",{"type":8,"value":2211,"toc":2559},[2212,2217,2219,2223,2226,2229,2232,2239,2242,2246,2249,2255,2269,2275,2281,2287,2293,2296,2300,2303,2307,2321,2324,2328,2342,2345,2349,2355,2358,2362,2365,2369,2372,2376,2379,2399,2402,2406,2409,2414,2428,2433,2444,2448,2459,2462,2466,2469,2475,2489,2493,2496,2500,2503,2506,2515,2518,2521,2523,2526,2529,2532,2535,2537,2540,2543,2546,2551],[11,2213,2214],{},[14,2215,2216],{},"Why real-time data matters, what makes migration hard, and how to think about the transition — whether you choose layline.io or another path",[23,2218],{},[26,2220,2222],{"id":2221},"the-batch-trap","The Batch Trap",[11,2224,2225],{},"There's a moment every data team eventually reaches. You've built cron jobs that run at 2 AM. Then another at 4. Then a third to clean up what the first two missed. Each job has its own schedule, its own dependencies, its own way of failing silently.",[11,2227,2228],{},"The original architect understood it all. But that person left two years ago. Now nobody touches the pipelines because nobody fully understands them — and nobody wants to be the one who breaks the overnight sync that feeds the entire reporting stack.",[11,2230,2231],{},"This is the batch trap. It sneaks up on you. Each individual job seems reasonable. But over time, you end up with a tangled web of overnight jobs, each adding latency to your data, each carrying the risk of silent failures that nobody notices until someone asks why the numbers look wrong.",[11,2233,2234,2235,2238],{},"Traditional ETL made sense when data freshness was a nice-to-have and reliability was everything. But the business world has changed. Customers expect instant notifications. Fraud teams need sub-second detection. Dashboards should show what's happening ",[14,2236,2237],{},"now",", not what happened yesterday.",[11,2240,2241],{},"If any of this sounds familiar, you're probably thinking about making the leap from batch to streaming. But how do you actually do it without breaking everything?",[26,2243,2245],{"id":2244},"the-real-challenges-of-moving-to-streaming","The Real Challenges of Moving to Streaming",[11,2247,2248],{},"Before we talk about solutions, let's be honest about what makes this migration difficult.",[11,2250,2251],{},[92,2252],{"alt":2253,"src":2254},"Batch vs Streaming Mental Model","/images/blog/etl-streaming-mental-model.jpg",[11,2256,2257,2260,2261,2264,2265,2268],{},[246,2258,2259],{},"The mental model shift is harder than the technical one."," Batch processing thinks in jobs and windows. Streaming thinks in events and continuous processing. If you try to port your batch logic directly to streaming, you'll fight the paradigm at every step. You need to rethink ",[14,2262,2263],{},"what"," triggers processing, not just ",[14,2266,2267],{},"how"," it's processed.",[11,2270,2271,2274],{},[246,2272,2273],{},"Stateful operations get complicated."," In batch, you load a table, do your join, write the result, and forget it. In streaming, that state lives in memory (or in a state store) and needs to be managed carefully. What happens when you restart? How do you handle late-arriving data?",[11,2276,2277,2280],{},[246,2278,2279],{},"Not everything migrates cleanly."," Some transformations that are trivial in batch — a massive join across two huge tables, for example — become expensive or impossible in pure streaming without rethinking the approach entirely.",[11,2282,2283,2286],{},[246,2284,2285],{},"The hybrid period is painful."," Unless you're building from scratch (rare), you'll run batch and streaming side-by-side during migration. This means double the infrastructure, double the monitoring, and the fun challenge of making sure both systems produce identical outputs.",[11,2288,2289,2292],{},[246,2290,2291],{},"Backpressure and exactly-once semantics"," are real engineering problems that don't exist in simple batch pipelines. When your Kafka topic suddenly gets 10x the traffic, your streaming system needs to handle it gracefully — not fall over.",[11,2294,2295],{},"These aren't insurmountable, but they're worth understanding before you start.",[26,2297,2299],{"id":2298},"approaches-to-the-problem","Approaches to the Problem",[11,2301,2302],{},"There's more than one way to solve this. Here are the main paths teams take:",[339,2304,2306],{"id":2305},"build-your-own-with-open-source-frameworks","Build Your Own with Open Source Frameworks",[11,2308,2309,2312,2313,2316,2317,2320],{},[246,2310,2311],{},"Apache Kafka"," + ",[246,2314,2315],{},"Apache Flink"," (or ",[246,2318,2319],{},"Spark Structured Streaming",") gives you maximum control. You can build exactly what you need. The tradeoff is infrastructure overhead: you're now operating two complex distributed systems, managing your own deployments, scaling, monitoring, and debugging when things go wrong.",[11,2322,2323],{},"This approach works well for teams with strong engineering resources who need fine-grained control over every aspect of their streaming infrastructure.",[339,2325,2327],{"id":2326},"go-all-in-on-a-managed-service","Go All-In on a Managed Service",[11,2329,2330,2333,2334,2337,2338,2341],{},[246,2331,2332],{},"AWS Kinesis Data Analytics",", ",[246,2335,2336],{},"Google Cloud Dataflow",", or ",[246,2339,2340],{},"Azure Stream Analytics"," handle the operational complexity for you. You focus on logic, not infrastructure.",[11,2343,2344],{},"The tradeoff is vendor lock-in. Once you build your pipelines in a managed service, migrating away becomes its own project. Cost can also be unpredictable at scale — these services can get expensive quickly.",[339,2346,2348],{"id":2347},"use-a-purpose-built-streaming-platform","Use a Purpose-Built Streaming Platform",[11,2350,2351,2352,2354],{},"Modern platforms like ",[246,2353,252],{}," sit between these two extremes. They give you visual tooling (reducing the coding burden) while staying infrastructure-agnostic — you can run on Kubernetes, in containers, or in the cloud of your choice.",[11,2356,2357],{},"The benefit is faster time-to-value: you don't need a team of distributed systems experts to get streaming pipelines into production. The consideration is evaluating whether the platform's abstraction level matches your needs.",[339,2359,2361],{"id":2360},"the-hybrid-path","The Hybrid Path",[11,2363,2364],{},"Most mature organizations don't do a wholesale migration. They run batch and streaming in parallel, gradually shifting high-value pipelines to real-time while keeping the batch safety net underneath. This is the reality for most teams — and it's okay.",[26,2366,2368],{"id":2367},"what-actually-works-a-migration-framework","What Actually Works: A Migration Framework",[11,2370,2371],{},"Regardless of which approach you choose, here's a practical framework that's emerged from teams who've done this successfully:",[339,2373,2375],{"id":2374},"start-with-inventory","Start with Inventory",[11,2377,2378],{},"Before you migrate anything, understand what you have:",[604,2380,2381,2387,2393],{},[362,2382,2383,2386],{},[246,2384,2385],{},"Map all ETL jobs"," — Identify their sources, transformations, and destinations",[362,2388,2389,2392],{},[246,2390,2391],{},"Classify by urgency"," — Which pipelines would benefit most from real-time? Start there.",[362,2394,2395,2398],{},[246,2396,2397],{},"Find the boundaries"," — Where does one job's output feed another's input?",[11,2400,2401],{},"This sounds basic, but most teams discover they have undocumented dependencies that only become visible when they try to change something.",[339,2403,2405],{"id":2404},"identify-what-migrates-cleanly","Identify What Migrates Cleanly",[11,2407,2408],{},"Not every transformation works equally well in streaming:",[11,2410,2411],{},[246,2412,2413],{},"Good streaming candidates:",[359,2415,2416,2419,2422,2425],{},[362,2417,2418],{},"Field-based filtering and routing",[362,2420,2421],{},"Enrichment with lookups (adding customer info to transactions)",[362,2423,2424],{},"Time-windowed aggregations (counts per minute, sums per hour)",[362,2426,2427],{},"Format conversions (JSON → Avro, XML → JSON)",[11,2429,2430],{},[246,2431,2432],{},"Needs rethinking:",[359,2434,2435,2438,2441],{},[362,2436,2437],{},"Large batch joins (may need stateful streaming joins)",[362,2439,2440],{},"Complex multi-step aggregations (break into smaller, composable steps)",[362,2442,2443],{},"Anything that assumes access to the \"full dataset\" at once",[339,2445,2447],{"id":2446},"design-for-events-not-jobs","Design for Events, Not Jobs",[11,2449,2450,2451,2454,2455,2458],{},"The biggest mental shift: think about what ",[14,2452,2453],{},"event"," should trigger processing, not what ",[14,2456,2457],{},"time"," should trigger processing. When a transaction occurs, enrich and route it immediately. Don't wait for midnight.",[11,2460,2461],{},"This changes how you think about completeness, too. In batch, you know when a window is \"done.\" In streaming, you need to think about watermark policies and late-data handling.",[339,2463,2465],{"id":2464},"plan-for-the-hybrid","Plan for the Hybrid",[11,2467,2468],{},"Expect to run both systems for a while:",[11,2470,2471],{},[92,2472],{"alt":2473,"src":2474},"Hybrid Batch and Streaming","/images/blog/etl-streaming-hybrid.jpg",[359,2476,2477,2480,2483,2486],{},[362,2478,2479],{},"Keep batch as a fallback during migration",[362,2481,2482],{},"Compare batch vs. streaming outputs using monitoring",[362,2484,2485],{},"Validate before cutting over",[362,2487,2488],{},"Accept that some pipelines might stay batch (if real-time isn't worth the effort)",[339,2490,2492],{"id":2491},"invest-in-observability-early","Invest in Observability Early",[11,2494,2495],{},"Whatever platform you choose, make sure you have good metrics from day one. Latency distributions, throughput, error rates, and processing backpressure — you need to see these at a glance.",[26,2497,2499],{"id":2498},"the-laylineio-angle","The Layline.io Angle",[11,2501,2502],{},"If you're evaluating purpose-built platforms for this transition, layline.io is worth a look. Here's what makes it different:",[11,2504,2505],{},"It uses a visual workflow designer, so your entire team can see and understand the data flow — not just whoever wrote the code. This matters when you're debugging at 2 AM or onboarding new team members.",[11,2507,2508,2509,2511,2512,2514],{},"It handles the operational bits — backpressure, state management, auto-scaling — without requiring you to become a distributed systems expert. You define ",[14,2510,2263],{}," processing should happen; the platform handles ",[14,2513,2267],{}," it runs reliably.",[11,2516,2517],{},"It stays infrastructure-agnostic: deploy on Kubernetes, Docker, or anywhere containers run. No vendor lock-in means you're not trapped if your requirements change.",[11,2519,2520],{},"For teams who want streaming capabilities without building a dedicated infrastructure team, this is the gap layline.io fills.",[23,2522],{},[26,2524,2525],{"id":197},"The Bottom Line",[11,2527,2528],{},"Moving from batch to streaming isn't really about rewriting your pipelines. It's about changing how you think about data: from snapshots in time to continuous flows.",[11,2530,2531],{},"Start with one high-value pipeline. Prove the pattern. Then expand.",[11,2533,2534],{},"Whether you build it yourself, go with a managed service, or use a platform like layline.io, the key is starting — and being honest about the tradeoffs along the way.",[23,2536],{},[26,2538,2539],{"id":212},"What's Next",[11,2541,2542],{},"If you're ready to explore streaming for your team, the best next step is understanding what your highest-value pipeline would be. Where would real-time data make the biggest impact?",[11,2544,2545],{},"For layline.io users, the Community Edition is free to try — no credit card required. You can build and deploy a simple streaming pipeline in an afternoon.",[11,2547,2548],{},[226,2549,2550],{"href":228},"Get Started with Community Edition →",[11,2552,2553],{},[14,2554,2555,2556],{},"Have a specific migration scenario? The team has helped dozens of teams make this transition. ",[226,2557,2558],{"href":1567},"Reach out →",{"title":255,"searchDepth":256,"depth":256,"links":2560},[2561,2562,2563,2569,2576,2577,2578],{"id":2221,"depth":256,"text":2222},{"id":2244,"depth":256,"text":2245},{"id":2298,"depth":256,"text":2299,"children":2564},[2565,2566,2567,2568],{"id":2305,"depth":540,"text":2306},{"id":2326,"depth":540,"text":2327},{"id":2347,"depth":540,"text":2348},{"id":2360,"depth":540,"text":2361},{"id":2367,"depth":256,"text":2368,"children":2570},[2571,2572,2573,2574,2575],{"id":2374,"depth":540,"text":2375},{"id":2404,"depth":540,"text":2405},{"id":2446,"depth":540,"text":2447},{"id":2464,"depth":540,"text":2465},{"id":2491,"depth":540,"text":2492},{"id":2498,"depth":256,"text":2499},{"id":197,"depth":256,"text":2525},{"id":212,"depth":256,"text":2539},"2026-03-17","/images/blog/2026-03-17/etl-to-streaming-hero.jpg",{},"/blog/2026-03-17-etl-to-streaming-migration-playbook",{"title":2209,"description":2216},{"loc":2582},"blog/2026-03-17-etl-to-streaming-migration-playbook","TJ0glWXJIczRLNcAtN-5LLmqQ4Pmaq9iKIvHwsvDb8Q",{"id":2588,"title":2589,"body":2590,"category":269,"date":2932,"description":2933,"extension":271,"featured":2934,"image":2935,"meta":2936,"navigation":272,"path":2937,"readTime":1605,"seo":2938,"sitemap":2939,"stem":2940,"__hash__":2941},"blog/blog/2026-02-15-serverless-hangover.md","Your Cloud Bill Called. It Wants a Word About Serverless.",{"type":8,"value":2591,"toc":2914},[2592,2595,2598,2605,2611,2615,2618,2638,2641,2648,2651,2655,2658,2662,2669,2672,2676,2679,2686,2690,2693,2699,2703,2706,2711,2714,2721,2724,2728,2731,2814,2820,2824,2833,2837,2840,2844,2851,2855,2866,2870,2873,2877,2884,2898,2901,2905,2911],[11,2593,2594],{},"Picture a restaurant kitchen. The head chef has just fired the old line cooks and replaced them with an army of on-demand temps. Every time an order comes in, a fresh temp materializes out of thin air, cooks exactly one dish, then vanishes. Genius, right? No idle wages. No wasted downtime.",[11,2596,2597],{},"Except some temps take ninety seconds just to find the frying pan. And the staffing agency's bill? It scales with every single plate.",[11,2599,2600,2601,2604],{},"That, in a nutshell, is the ",[246,2602,2603],{},"serverless trade-off"," that engineering teams across the industry are now reckoning with.",[11,2606,2607],{},[92,2608],{"alt":2609,"src":2610},"A cartoon kitchen where confused chefs appear in puffs of smoke while a manager stares at an enormous bill","/images/blog/2026-02-15/kitchen-chaos.jpg",[26,2612,2614],{"id":2613},"the-pitch-vs-the-invoice","The Pitch vs. The Invoice",[11,2616,2617],{},"The serverless sales pitch is seductive:",[359,2619,2620,2626,2632],{},[362,2621,2622,2625],{},[246,2623,2624],{},"Zero provisioning."," No servers to patch at 2 AM.",[362,2627,2628,2631],{},[246,2629,2630],{},"Elastic scaling."," Sudden traffic spike? Handled.",[362,2633,2634,2637],{},[246,2635,2636],{},"Pay-per-invocation."," Idle means free.",[11,2639,2640],{},"For certain use cases—an occasional webhook, a nightly ETL job, a one-off image resize—this model is genuinely brilliant. It's cloud computing distilled to its purest form.",[11,2642,2643,2644,2647],{},"But many teams didn't stop there. They took the model and applied it to ",[246,2645,2646],{},"everything",": authentication flows, order pipelines, payment processing, real-time analytics. What started as \"let's simplify\" became a constellation of hundreds of tiny functions, each invisible, each independently billed, and each holding a small piece of the puzzle that no single engineer could see in full.",[11,2649,2650],{},"The architecture slide looked elegant. The monthly cost spreadsheet did not.",[26,2652,2654],{"id":2653},"three-cracks-in-the-façade","Three Cracks in the Façade",[11,2656,2657],{},"When workloads shift from \"occasional bursts\" to \"steady streams,\" the cracks appear fast.",[339,2659,2661],{"id":2660},"the-wake-up-tax","The Wake-Up Tax",[11,2663,2664,2665,2668],{},"Every ephemeral function has a boot sequence. That boot sequence has a cost—not in dollars, but in ",[246,2666,2667],{},"milliseconds",". For a background job, nobody notices. For a user staring at a checkout button, those extra hundreds of milliseconds feel like an eternity. Multiply that across a chain of three or four functions in sequence and tail latency balloons into something your SLA cannot absorb.",[11,2670,2671],{},"Think of it like a relay race where every runner has to tie their shoes before sprinting. Average pace looks fine on paper. But the audience only remembers the handoff where someone fumbled.",[339,2673,2675],{"id":2674},"the-observability-maze","The Observability Maze",[11,2677,2678],{},"When a request touches a single process, tracing it is trivial. When that same request fans out across a gateway, an auth function, a business-logic function, a notification function, and a persistence function—each hosted by the cloud provider in its own sandbox—debugging becomes archaeology.",[11,2680,2681,2682,2685],{},"Logs scatter across consoles. Metrics live in separate dashboards. Correlating a slow response means stitching together breadcrumbs from five different vendor UIs. Engineers spend less time fixing bugs and more time ",[246,2683,2684],{},"finding"," them.",[339,2687,2689],{"id":2688},"the-invisible-ceiling","The Invisible Ceiling",[11,2691,2692],{},"Cloud providers enforce per-function concurrency limits, regional quotas, and connection caps that are easy to overlook during development. Under real load, these limits surface as throttled requests, dropped connections, or mysterious 429 errors. The worst part? You usually discover them at the exact moment you can least afford surprises—during a traffic peak.",[11,2694,2695],{},[92,2696],{"alt":2697,"src":2698},"A wall of browser tabs and dashboards representing the observability nightmare of debugging serverless","/images/blog/2026-02-15/observability-maze.jpg",[26,2700,2702],{"id":2701},"steady-rivers-dont-need-rain-dances","Steady Rivers Don't Need Rain Dances",[11,2704,2705],{},"Here's the mental model that clarifies everything:",[11,2707,2708],{},[246,2709,2710],{},"Serverless is optimized for storms. Most production workloads are rivers.",[11,2712,2713],{},"A storm is unpredictable, short-lived, and violent. You want elastic capacity that appears and disappears. Functions excel here.",[11,2715,2716,2717,2720],{},"A river is continuous, predictable, and relentless. It flows during business hours, it flows overnight, it flows on weekends. For rivers, you don't need magic elasticity. You need a ",[246,2718,2719],{},"channel","—something persistent, well-shaped, and always ready.",[11,2722,2723],{},"Data processing workloads almost always behave like rivers. Telco CDRs arrive every second. Financial transactions tick constantly. IoT sensors never sleep. Feeding these streams through ephemeral functions is like routing a river through a series of pop-up tents. It technically works. But you'll spend all your time rebuilding tents.",[26,2725,2727],{"id":2726},"what-the-numbers-actually-say","What the Numbers Actually Say",[11,2729,2730],{},"Teams that have compared both approaches for steady-state workloads consistently find a pattern:",[1958,2732,2733,2747],{},[1961,2734,2735],{},[1964,2736,2737,2740,2744],{},[1967,2738,1969],{"align":2739},"left",[1967,2741,2743],{"align":2742},"center","Ephemeral Functions",[1967,2745,2746],{"align":2742},"Persistent Engine",[1977,2748,2749,2762,2775,2788,2801],{},[1964,2750,2751,2756,2759],{},[1982,2752,2753],{"align":2739},[246,2754,2755],{},"p95 Latency",[1982,2757,2758],{"align":2742},"Variable (cold-start spikes)",[1982,2760,2761],{"align":2742},"Stable and low",[1964,2763,2764,2769,2772],{},[1982,2765,2766],{"align":2739},[246,2767,2768],{},"Cost at Low Volume",[1982,2770,2771],{"align":2742},"Lower",[1982,2773,2774],{"align":2742},"Higher",[1964,2776,2777,2782,2785],{},[1982,2778,2779],{"align":2739},[246,2780,2781],{},"Cost at Sustained Volume",[1982,2783,2784],{"align":2742},"Significantly higher",[1982,2786,2787],{"align":2742},"Significantly lower",[1964,2789,2790,2795,2798],{},[1982,2791,2792],{"align":2739},[246,2793,2794],{},"Debugging Effort",[1982,2796,2797],{"align":2742},"High (distributed traces)",[1982,2799,2800],{"align":2742},"Low (single process)",[1964,2802,2803,2808,2811],{},[1982,2804,2805],{"align":2739},[246,2806,2807],{},"Onboarding Complexity",[1982,2809,2810],{"align":2742},"Many moving parts to learn",[1982,2812,2813],{"align":2742},"One system to understand",[11,2815,2816,2817,1515],{},"The crossover point arrives faster than most teams expect. Once your baseline traffic is steady, the per-invocation billing model stops being a bargain and starts being a ",[246,2818,2819],{},"multiplier",[26,2821,2823],{"id":2822},"how-laylineio-turns-the-river-into-a-pipeline","How layline.io Turns the River Into a Pipeline",[11,2825,2826,2827,2829,2830,1515],{},"This is precisely the problem ",[246,2828,252],{}," was designed to solve. Instead of scattering your data logic across dozens of ephemeral functions stitched together with YAML and hope, layline.io gives you a ",[246,2831,2832],{},"persistent, visual, always-hot data engine",[339,2834,2836],{"id":2835},"always-running-always-ready","Always Running, Always Ready",[11,2838,2839],{},"layline.io deploys as a long-lived service on your own infrastructure—VMs, Kubernetes, bare metal, your call. There are no cold starts because the engine never sleeps. Data arrives, gets processed, and moves on. No boot tax. No wake-up jitter.",[339,2841,2843],{"id":2842},"one-canvas-not-fifty-consoles","One Canvas, Not Fifty Consoles",[11,2845,2846,2847,2850],{},"With layline.io's ",[246,2848,2849],{},"drag-and-drop workflow designer",", your entire data pipeline lives on a single visual canvas. Sources (Kafka, HTTP, files, databases), transformations, routing logic, and destinations—all visible in one place. When something goes wrong, you don't need a treasure map. You click on the step and look.",[339,2852,2854],{"id":2853},"built-for-pressure","Built for Pressure",[11,2856,2857,2858,2861,2862,2865],{},"Under the hood, layline.io is powered by the ",[246,2859,2860],{},"Apache Pekko"," framework—the same actor-model technology trusted by some of the world's highest-throughput systems. It handles ",[246,2863,2864],{},"back-pressure"," natively, meaning that when downstream systems slow down, layline.io throttles gracefully instead of dropping messages or crashing. Guaranteed processing, not best-effort.",[339,2867,2869],{"id":2868},"predictable-economics","Predictable Economics",[11,2871,2872],{},"No per-invocation billing surprises. You provision the capacity you need, layline.io utilizes it efficiently, and your finance team can finally forecast infrastructure costs without a ouija board.",[26,2874,2876],{"id":2875},"the-pragmatic-split","The Pragmatic Split",[11,2878,2879,2880,2883],{},"None of this means serverless should be banished entirely. The smart move is a ",[246,2881,2882],{},"division of labor",":",[359,2885,2886,2892],{},[362,2887,2888,2891],{},[246,2889,2890],{},"Serverless"," for the edges: occasional triggers, lightweight glue, spike-driven background tasks.",[362,2893,2894,2897],{},[246,2895,2896],{},"A persistent engine like layline.io"," for the core: steady data flows, real-time processing, mission-critical pipelines that run all day and pay the bills.",[11,2899,2900],{},"This isn't about ideology. It's about matching the tool to the workload. A hammer is great—until you need a wrench.",[26,2902,2904],{"id":2903},"the-takeaway","The Takeaway",[11,2906,2907,2908,1515],{},"If your cloud bill grows faster than your traffic, if debugging a single slow request takes longer than fixing it, or if your team spends more time wrestling platform quirks than shipping features—",[246,2909,2910],{},"the architecture is fighting the workload",[11,2912,2913],{},"Stop routing rivers through pop-up tents.\nBuild a proper pipeline.",{"title":255,"searchDepth":256,"depth":256,"links":2915},[2916,2917,2922,2923,2924,2930,2931],{"id":2613,"depth":256,"text":2614},{"id":2653,"depth":256,"text":2654,"children":2918},[2919,2920,2921],{"id":2660,"depth":540,"text":2661},{"id":2674,"depth":540,"text":2675},{"id":2688,"depth":540,"text":2689},{"id":2701,"depth":256,"text":2702},{"id":2726,"depth":256,"text":2727},{"id":2822,"depth":256,"text":2823,"children":2925},[2926,2927,2928,2929],{"id":2835,"depth":540,"text":2836},{"id":2842,"depth":540,"text":2843},{"id":2853,"depth":540,"text":2854},{"id":2868,"depth":540,"text":2869},{"id":2875,"depth":256,"text":2876},{"id":2903,"depth":256,"text":2904},"2026-02-15","From invisible scaling to invisible invoices—why engineering teams are ditching FaaS for persistent, predictable data engines.",false,"/images/blog/2026-02-15/serverless-hangover.jpg",{},"/blog/2026-02-15-serverless-hangover",{"title":2589,"description":2933},{"loc":2937},"blog/2026-02-15-serverless-hangover","JpL075fo7f7gL4NgXrmg6UhWZrv6z-f3pzqeh6RGHLE",{"id":2943,"title":2944,"body":2945,"category":269,"date":3122,"description":3123,"extension":271,"featured":2934,"image":3124,"meta":3125,"navigation":272,"path":3126,"readTime":3127,"seo":3128,"sitemap":3129,"stem":3130,"__hash__":3131},"blog/blog/2024-06-30-apache-pekko.md","Discover the Power of Apache Pekko with layline.io",{"type":8,"value":2946,"toc":3114},[2947,2951,2954,2958,2961,2965,2997,3001,3004,3007,3010,3014,3076,3080,3083],[26,2948,2950],{"id":2949},"what-is-apache-pekko","What is Apache Pekko?",[11,2952,2953],{},"Apache Pekko is a powerful open-source message platform that enables developers to build scalable, secure, and high-performance data processing scenarios with minimal coding. It provides a wide range of features and tools to streamline the development process and accelerate time-to-market. It is widely used in various industries, such as finance, healthcare, e-commerce, and more, to create custom applications that meet specific business requirements.",[26,2955,2957],{"id":2956},"why-apache-pekko-is-a-game-changer","Why Apache Pekko is a Game-Changer",[11,2959,2960],{},"The framework offers several benefits that make it a game-changer in the world of data processing. Here are some key features that set it apart from other frameworks:",[26,2962,2964],{"id":2963},"key-benefits-of-apache-pekko","Key Benefits of Apache Pekko",[604,2966,2967,2973,2979,2985,2991],{},[362,2968,2969,2972],{},[246,2970,2971],{},"Scalability",": Apache Pekko is designed to scale horizontally and vertically to meet the demands of large-scale applications. It can handle high volumes of data and traffic, making it ideal for mission-critical applications that require high availability and performance.",[362,2974,2975,2978],{},[246,2976,2977],{},"Resilience",": Built with fault tolerance in mind, Apache Pekko ensures that your applications are resilient to failures. By leveraging the actor model, it isolates failures and recovers from them gracefully, keeping your system running smoothly even in the face of unexpected issues.",[362,2980,2981,2984],{},[246,2982,2983],{},"Concurrency",": Concurrency is at the core of Apache Pekko. It allows you to build applications that can handle multiple tasks simultaneously without the typical challenges of thread management. This results in highly efficient and responsive applications.",[362,2986,2987,2990],{},[246,2988,2989],{},"Message-Driven Architecture",": It's message-driven architecture promotes loose coupling, location transparency, and high performance. It simplifies the complexity of building distributed systems, enabling you to focus on your business logic rather than infrastructure concerns.",[362,2992,2993,2996],{},[246,2994,2995],{},"Flexibility",": Whether you're building microservices, streaming data applications, or real-time analytics platforms, Apache Pekko's versatile toolkit supports a wide range of use cases. Its modular architecture allows you to use only the components you need, making it adaptable to various application requirements.",[26,2998,3000],{"id":2999},"the-downside-of-apache-pekko","The Downside of Apache Pekko",[11,3002,3003],{},"While Pekko is a truly awesome framework, it is not easy to use. It requires a deep understanding of its concepts and a significant amount of coding to build and deploy applications. This complexity can be a barrier for many developers who want to leverage the power of Apache Pekko but lack the expertise to do so. It also lacks a user-friendly interface and deployment and monitoring tools, making it challenging to build and manage applications effectively.",[11,3005,3006],{},"To overcome these challenges, layline.io has developed a low-code platform that abstracts the complexities of Apache Pekko and provides a visual interface for designing, deploying, and monitoring applications. With layline.io, you can harness the power of Apache Pekko without writing extensive code, enabling you to build scalable, resilient, and high-performance applications with ease.",[11,3008,3009],{},"Of course, there is also much, much more to layline.io than just the abstraction of Apache Pekko. Some of which you can explore in the following sections.",[26,3011,3013],{"id":3012},"laylineio-bringing-apache-pekko-to-everyone","layline.io: Bringing Apache Pekko to Everyone",[604,3015,3016,3022,3028,3034,3040,3046,3052,3058,3064,3070],{},[362,3017,3018,3021],{},[246,3019,3020],{},"Simplified Development",": With layline.io, you don't need to write intricate code to harness the power of Apache Pekko. Our intuitive visual interface allows you to design and deploy applications rapidly, reducing development time and accelerating your time-to-market.",[362,3023,3024,3027],{},[246,3025,3026],{},"Drag-and-Drop Interface",": Our user-friendly drag-and-drop interface lets you build complex workflows effortlessly. Create Workflows, define data structures and individual processing logic, and set up your entire application flow without delving into the underlying codebase.",[362,3029,3030,3033],{},[246,3031,3032],{},"One-click deployment",": Deploy your project's workflows in full or selectively to a layline.io Reactive cluster with just one click. No hassle with manual configurations or complex setup procedures. layline.io takes care of the deployment process, so you can focus on building great applications.",[362,3035,3036,3039],{},[246,3037,3038],{},"Real-time Monitoring",": Monitor your application's performance in real-time with layline.io's built-in monitoring tools. Keep track of data processing, workflow execution, and system health to ensure optimal performance and reliability.",[362,3041,3042,3045],{},[246,3043,3044],{},"Scalability and Resilience",": layline.io leverages Apache Pekko's scalability and resilience to provide you with a robust platform for building mission-critical applications. Whether you're processing real-time data streams or managing complex workflows, layline.io ensures your applications are scalable, reliable, and high-performing.",[362,3047,3048,3051],{},[246,3049,3050],{},"Cloud-Native Architecture",": Built on a cloud-native architecture, layline.io enables you to deploy your applications in the cloud with ease. Take advantage of cloud scalability, elasticity, and cost-efficiency to run your applications in a modern, cloud-native environment.",[362,3053,3054,3057],{},[246,3055,3056],{},"Low-Code Development",": With layline.io, you can build sophisticated applications without writing extensive code. Our low-code platform abstracts the complexities of Apache Pekko, allowing you to focus on designing innovative solutions and delivering value to your users.",[362,3059,3060,3063],{},[246,3061,3062],{},"Seamless Integration",": Integrate with your existing systems and third-party services seamlessly. layline.io supports a wide range of connectors, enabling you to connect to databases, messaging queues, APIs, and more with just a few clicks.",[362,3065,3066,3069],{},[246,3067,3068],{},"Comprehensive Security",": Security is a top priority at layline.io. We provide robust security features to protect your data, applications, and infrastructure. From encryption to access controls, we ensure that your applications are secure and compliant with industry standards.",[362,3071,3072,3075],{},[246,3073,3074],{},"Comprehensive Support",": Our team of experts is here to support you at every step of your application development journey. From onboarding to deployment, we provide comprehensive support to help you build, deploy, and manage your applications effectively.",[26,3077,3079],{"id":3078},"get-started-with-laylineio-today","Get Started with layline.io Today",[11,3081,3082],{},"Experience the power of Apache Pekko without the complexity. Join the growing community of developers who are transforming their application development process with layline.io. Sign up for a free trial today and see how easy it is to build scalable, resilient, and high-performance applications with our low-code platform.",[359,3084,3085,3093,3100,3107],{},[362,3086,3087,3088,1515],{},"Read more about layline.io ",[226,3089,3092],{"href":3090,"rel":3091},"https://layline.io/",[2187],"here",[362,3094,3095,3096,1515],{},"Get started with layline.io ",[226,3097,3092],{"href":3098,"rel":3099},"https://layline.io/download",[2187],[362,3101,3102,3103,1515],{},"Read more about Apache Pekko ",[226,3104,3092],{"href":3105,"rel":3106},"https://pekko.apache.org/",[2187],[362,3108,3109,3110,1515],{},"Contact us at ",[226,3111,3113],{"href":3112},"mailto:hello@layline.io","hello@layline.io",{"title":255,"searchDepth":256,"depth":256,"links":3115},[3116,3117,3118,3119,3120,3121],{"id":2949,"depth":256,"text":2950},{"id":2956,"depth":256,"text":2957},{"id":2963,"depth":256,"text":2964},{"id":2999,"depth":256,"text":3000},{"id":3012,"depth":256,"text":3013},{"id":3078,"depth":256,"text":3079},"2024-06-30","At layline.io, we've harnessed the robust capabilities of Apache Pekko to bring you a comprehensive low-code event-processing platform. With our solution, you can leverage the full potential of Apache Pekko without writing a single line of code.","/images/blog/2024-06-30/pekko_logo-h300.png",{},"/blog/2024-06-30-apache-pekko","10 min",{"title":2944,"description":3123},{"loc":3126},"blog/2024-06-30-apache-pekko","t4jZ0J_Sj5b7Zz8t-rIkqncLMQAB2VeQ6gXVTNxQmgg",{"id":3133,"title":3134,"body":3135,"category":3329,"date":3330,"description":3331,"extension":271,"featured":2934,"image":3332,"meta":3333,"navigation":272,"path":3334,"readTime":553,"seo":3335,"sitemap":3336,"stem":3337,"__hash__":3338},"blog/blog/2023-10-16-ubiquity-unifi-showcase.md","The Power of Data Orchestration with layline.io: A Ubiquity Unifi Showcase Scenario",{"type":8,"value":3136,"toc":3311},[3137,3140,3144,3147,3150,3154,3157,3163,3167,3170,3176,3180,3183,3189,3193,3196,3202,3206,3209,3215,3219,3222,3228,3232,3235,3239,3242,3246,3249,3253,3256,3260,3263,3267,3270,3274,3277,3280],[11,3138,3139],{},"Today, we're going to dive into the fascinating world of data orchestration and how layline.io, a cutting-edge software, can empower you to streamline your data management needs. We'll also walk you through a showcase scenario to demonstrate how layline.io can simplify even the most complex data handling tasks.",[26,3141,3143],{"id":3142},"the-birth-of-laylineio","The Birth of layline.io",[11,3145,3146],{},"layline.io is a software solution designed to handle data orchestration in a fully configurable and customizable manner. Created to cater to a wide range of industries and use cases, layline.io provides users with the ability to manage data flow, automate processes, and transform data from one format to another seamlessly.",[11,3148,3149],{},"The software's architecture is built around the idea that users should have complete control over their data workflows. Whether you're dealing with data extraction, transformation, or loading tasks, layline.io offers a user-friendly interface for defining, configuring, and executing data pipelines. The system operates with the flexibility of a swiss-army knife, letting you mold it to fit your specific needs.",[26,3151,3153],{"id":3152},"a-showcase-scenario-data-management-for-a-unifi-network-controller","A Showcase Scenario: Data Management for a Unifi Network Controller",[11,3155,3156],{},"Let's delve into a specific use case to illustrate the power of layline.io. Imagine you're an IT administrator managing a Unifi network controller, responsible for ensuring smooth operations and troubleshooting network issues. In this scenario, layline.io can be configured to automate the following tasks:",[11,3158,3159],{},[92,3160],{"alt":3161,"src":3162},"Ubiquity Workflow","/images/blog/2023-10-16/ubiquity-workflow.png",[339,3164,3166],{"id":3165},"_1-login-to-a-unifi-network-controller","1. Login to a Unifi Network Controller",[11,3168,3169],{},"layline.io's configurable interface allows you to set up an automated login process to the Unifi network controller. This ensures secure access to network data.",[11,3171,3172],{},[92,3173],{"alt":3174,"src":3175},"Ubiquity Login","/images/blog/2023-10-16/ubiquity-login.png",[339,3177,3179],{"id":3178},"_2-retrieve-client-status-information","2. Retrieve Client Status Information",[11,3181,3182],{},"Once logged in, layline.io initiates HTTP REST calls to fetch status information for all network-connected clients. This might include data on device health, signal strength, activity, and more.",[11,3184,3185],{},[92,3186],{"alt":3187,"src":3188},"Get Client States","/images/blog/2023-10-16/ubiquity-allstatus.png",[339,3190,3192],{"id":3191},"_3-format-data-into-csv","3. Format Data into CSV",[11,3194,3195],{},"With the data in hand, layline.io provides the tools to transform and format the information into CSV, a widely recognized and accessible data format. Any other format is also possible, simply by configuration.",[11,3197,3198],{},[92,3199],{"alt":3200,"src":3201},"Data output format","/images/blog/2023-10-16/ubiquity-output-format.png",[339,3203,3205],{"id":3204},"_4-output-data-to-a-file","4. Output Data to a File",[11,3207,3208],{},"Having the formatted data ready, layline.io can be configured to write the information to a file, which can be easily shared or integrated into other systems.",[11,3210,3211],{},[92,3212],{"alt":3213,"src":3214},"Files written","/images/blog/2023-10-16/ubiquity-files.png",[339,3216,3218],{"id":3217},"_5-automated-and-timely-execution","5. Automated and Timely Execution",[11,3220,3221],{},"The entire process can be automatically repeated at specific intervals. In this scenario, the data retrieval and transformation tasks can be set to occur every 15 seconds, ensuring that the IT administrator always has access to real-time network information.",[11,3223,3224],{},[92,3225],{"alt":3226,"src":3227},"Timer configuration","/images/blog/2023-10-16/ubiquity-timer.png",[339,3229,3231],{"id":3230},"_6-feed-data-into-monitoring-systems","6. Feed Data into Monitoring Systems",[11,3233,3234],{},"Now, here's where the magic happens. Data generated by layline.io can be seamlessly integrated into monitoring systems like Nagios, Prometheus, Grafana, Zabbix, or custom-built solutions. A direct feed instead of a file is supported, of course. This allows for real-time tracking of network health and client activity, with the added bonus of instant alerts triggered by predefined thresholds or anomalies.",[26,3236,3238],{"id":3237},"the-benefits-of-laylineio","The Benefits of layline.io",[11,3240,3241],{},"By implementing layline.io in this showcase scenario, we can see several key advantages:",[339,3243,3245],{"id":3244},"_1-efficiency-and-accuracy","1. Efficiency and Accuracy",[11,3247,3248],{},"Automating routine data tasks ensures that the process is carried out consistently and without errors. This significantly reduces the risk of human error and enhances the efficiency of data management.",[339,3250,3252],{"id":3251},"_2-real-time-insights","2. Real-time Insights",[11,3254,3255],{},"The ability to perform data orchestration tasks at regular intervals provides real-time insights, which are crucial in the fast-paced world of IT management.",[339,3257,3259],{"id":3258},"_3-customization-and-flexibility","3. Customization and Flexibility",[11,3261,3262],{},"layline.io's highly configurable nature ensures it can be adapted to fit various use cases and industries, making it a valuable tool for a wide range of professionals.",[339,3264,3266],{"id":3265},"_4-data-security","4. Data Security",[11,3268,3269],{},"layline.io's secure login process ensures that sensitive data remains protected, making it a trusted choice for organizations that prioritize data security.",[26,3271,3273],{"id":3272},"conclusion","Conclusion",[11,3275,3276],{},"layline.io is not just a tool; it's a strategic partner in your journey to mastering data. In this blog post, we've seen how it can simplify and enhance data management while enabling real-time network monitoring and alerting. The integration of layline.io with monitoring systems transforms it into an indispensable asset for businesses aiming to stay ahead in today's data-driven world. It empowers you to take control of your data, streamline operations, and respond proactively to emerging challenges.",[11,3278,3279],{},"With layline.io, data becomes more than just information—it becomes a catalyst for progress and success. Moreover, the power of layline.io extends far beyond network client status information; it can extract and analyze a wealth of other data from virtually any network data source, including system logs, site events, alarms, configuration data, and much more, making it a comprehensive solution for holistic network management and optimization.",[359,3281,3282,3288,3295,3302,3307],{},[362,3283,3284,3285,1515],{},"Download layline.io ",[226,3286,3092],{"href":3098,"rel":3287},[2187],[362,3289,3290,3291,1515],{},"Download layline.io Docker Container ",[226,3292,3092],{"href":3293,"rel":3294},"https://hub.docker.com/r/layline/layline-samples",[2187],[362,3296,3297,3298,1515],{},"Download the sample project from GitHub ",[226,3299,3092],{"href":3300,"rel":3301},"https://github.com/layline-io-sample-projects/sample-http-ubiquity",[2187],[362,3303,3087,3304,1515],{},[226,3305,3092],{"href":3090,"rel":3306},[2187],[362,3308,3109,3309,1515],{},[226,3310,3113],{"href":3112},{"title":255,"searchDepth":256,"depth":256,"links":3312},[3313,3314,3322,3328],{"id":3142,"depth":256,"text":3143},{"id":3152,"depth":256,"text":3153,"children":3315},[3316,3317,3318,3319,3320,3321],{"id":3165,"depth":540,"text":3166},{"id":3178,"depth":540,"text":3179},{"id":3191,"depth":540,"text":3192},{"id":3204,"depth":540,"text":3205},{"id":3217,"depth":540,"text":3218},{"id":3230,"depth":540,"text":3231},{"id":3237,"depth":256,"text":3238,"children":3323},[3324,3325,3326,3327],{"id":3244,"depth":540,"text":3245},{"id":3251,"depth":540,"text":3252},{"id":3258,"depth":540,"text":3259},{"id":3265,"depth":540,"text":3266},{"id":3272,"depth":256,"text":3273},"Tutorial","2023-10-16","In an age where data rules supreme, managing and orchestrating the vast sea of information has become the backbone of numerous businesses and industries.","/images/blog/2023-10-16/2023-10-16-ubiquity.png",{},"/blog/2023-10-16-ubiquity-unifi-showcase",{"title":3134,"description":3331},{"loc":3334},"blog/2023-10-16-ubiquity-unifi-showcase","xRQ9z_b8tqSHHi5wHz6CsgzA46H2onOdvHqQBSxwjnE",{"id":3340,"title":3341,"body":3342,"category":269,"date":3455,"description":3456,"extension":271,"featured":2934,"image":3457,"meta":3458,"navigation":272,"path":3459,"readTime":1605,"seo":3460,"sitemap":3461,"stem":3462,"__hash__":3463},"blog/blog/2023-10-06-hamburg-startup-scene.md","layline.io: Revolutionizing Hamburg's IT Startup Scene with Event Data Processing Expertise",{"type":8,"value":3343,"toc":3447},[3344,3347,3351,3354,3358,3383,3387,3406,3410,3419,3423,3426,3428,3431],[11,3345,3346],{},"Greetings, dear readers! We are elated to share some exciting news with you. Recently, Eustartup.news recognized layline.io as one of the most influential IT startups in Hamburg, Germany. Today, we'll delve deeper into what makes layline.io a standout player in Hamburg's vibrant startup ecosystem, focusing specifically on our groundbreaking event-data-processing platform.",[26,3348,3350],{"id":3349},"laylineio-unleashing-the-power-of-event-data-processing","layline.io: Unleashing the Power of Event Data Processing",[11,3352,3353],{},"Let's start with the big question: What exactly is layline.io? We are a pioneering event-data-processing platform that has redefined the way businesses harness the potential of real-time and batch event data. Our platform is engineered to seamlessly handle massive volumes of event data, all while providing a high degree of configurability and operational flexibility.",[26,3355,3357],{"id":3356},"key-features-of-laylineio","Key Features of layline.io",[604,3359,3360,3366,3372,3377],{},[362,3361,3362,3365],{},[246,3363,3364],{},"Reactive Stream Management",": At the heart of layline.io is the power of reactive stream management. We empower businesses to manage and process event data in real-time with unparalleled efficiency.",[362,3367,3368,3371],{},[246,3369,3370],{},"Configurability",": Our platform allows you to define individual workflows and business logic through simple configurations, eliminating the need for custom object coding. This streamlines the process of adapting to changing data formats and sources.",[362,3373,3374,3376],{},[246,3375,2971],{},": Whether you're dealing with local event data or require fully distributed and resilient processing, layline.io has you covered. Our platform can scale effortlessly to meet your needs.",[362,3378,3379,3382],{},[246,3380,3381],{},"Data Agnosticism",": No matter the data format, source, or destination, layline.io can seamlessly handle it all. We empower businesses to deal with any data scenario with ease.",[26,3384,3386],{"id":3385},"why-choose-laylineio","Why Choose layline.io?",[604,3388,3389,3395,3400],{},[362,3390,3391,3394],{},[246,3392,3393],{},"Efficiency",": Our platform optimizes event-data processing, enabling you to make informed decisions in real time.",[362,3396,3397,3399],{},[246,3398,2995],{},": Tailor workflows and business logic to suit your specific requirements without the need for custom coding.",[362,3401,3402,3405],{},[246,3403,3404],{},"Simplicity",": Say goodbye to complex data handling. Let layline.io simplify the process for you.",[26,3407,3409],{"id":3408},"the-recognition","The Recognition",[11,3411,3412,3413,3418],{},"Eustartup.news, a respected online magazine, recently highlighted layline.io in their article titled \"",[226,3414,3417],{"href":3415,"rel":3416},"https://eustartup.news/which-are-the-most-influential-it-startups-in-hamburg-today/",[2187],"Which Are the Most Influential IT Startups in Hamburg Today?","\" Being featured among the top startups in Hamburg validates our commitment to innovation and excellence in the event-data-processing space.",[26,3420,3422],{"id":3421},"join-us-on-our-journey","Join Us on Our Journey",[11,3424,3425],{},"Our journey is fueled by a passion for simplifying event-data processing and enabling businesses to make data-driven decisions effortlessly. If your business deals with event data and you're seeking a game-changing solution, we invite you to explore layline.io. Experience the power of our platform and discover how it can revolutionize the way you handle real-time and batch event data.",[26,3427,3273],{"id":3272},[11,3429,3430],{},"We're immensely proud to be recognized as one of the most influential startups in Hamburg. This acknowledgment motivates us to continue pushing the boundaries of event-data processing. Thank you to our valued clients, partners, and the entire Hamburg startup community for their trust and support. Together, we'll continue to redefine the landscape of event-data processing, making it simpler and more powerful for businesses in Hamburg and beyond. Thank you for joining us on this thrilling journey!",[359,3432,3433,3438,3443],{},[362,3434,3087,3435,1515],{},[226,3436,3092],{"href":3090,"rel":3437},[2187],[362,3439,3440],{},[226,3441,3417],{"href":3415,"rel":3442},[2187],[362,3444,3109,3445,1515],{},[226,3446,3113],{"href":3112},{"title":255,"searchDepth":256,"depth":256,"links":3448},[3449,3450,3451,3452,3453,3454],{"id":3349,"depth":256,"text":3350},{"id":3356,"depth":256,"text":3357},{"id":3385,"depth":256,"text":3386},{"id":3408,"depth":256,"text":3409},{"id":3421,"depth":256,"text":3422},{"id":3272,"depth":256,"text":3273},"2023-10-06","layline.io selected one of the most influential IT-Startups in Hamburg, Germany by EU Startup News.","/images/blog/2023-10-06/image_hamburg_2023-10-06.jpg",{},"/blog/2023-10-06-hamburg-startup-scene",{"title":3341,"description":3456},{"loc":3459},"blog/2023-10-06-hamburg-startup-scene","oQZWUpVWdLAWdrk493ik4q6hzj5gEyd3F9RSQYuloSI",{"id":3465,"title":3466,"body":3467,"category":269,"date":3533,"description":3534,"extension":271,"featured":2934,"image":3535,"meta":3536,"navigation":272,"path":3537,"readTime":553,"seo":3538,"sitemap":3539,"stem":3540,"__hash__":3541},"blog/blog/2023-05-21-h-hotels-digital-transformation.md","H-Hotels.com - Digital Transformation with layline.io",{"type":8,"value":3468,"toc":3530},[3469,3472,3475,3478,3481,3484,3487,3490,3493,3499,3502,3507,3509,3512],[11,3470,3471],{},"In this blog post, we'll explore the success story of layline.io at H-Hotels.com, a German hotel chain with more than 60 hotels in Germany, Austria, Switzerland, and Hungary, that uses layline.io to help with their digital transformation journey.",[11,3473,3474],{},"H-Hotels.com has always been a trailblazer when it comes to staying ahead of the curve regarding business intelligence and automation. In this evolution, a conglomerate of custom-made processes started piling up to deal with the ever-increasing number of data sources and sinks as necessary.",[11,3476,3477],{},"Naturally, the challenges of keeping up with data integration tasks have increased over time. Some of which included responsiveness to get new data streams rolling, leveraging better and more data for customer insights, and perform necessary improvements in tracking and managing their inventory. The H-Hotels group knew that it needed to make some changes to stay ahead of the curve and decided to further explore the possibilities of data integration software.",[11,3479,3480],{},"After careful research and evaluation, H-Hotels.com chose layline.io, which offers the company a comprehensive platform to streamline its operations, manage its inventory and gain valuable insights into its customers. With the help of our team, H-Hotels.com was able to set up the software, integrate their existing systems, and begin collecting operations and customer data from all hotels in a matter of days.",[11,3482,3483],{},"One of the most significant benefits of using layline.io was that it allowed the H-Hotels group to gain an even more comprehensive view of its operations. The company was able to better understand how its rooms were being used, which amenities were most popular, and which marketing campaigns were driving the most bookings. This information allowed H-Hotels.com to make data-driven decisions about how to allocate resources and improve their services, resulting in increased revenue and customer satisfaction.",[11,3485,3486],{},"Another key advantage of putting layline.io to work is that it enables the H-Hotels group to automate many of its processes. Things that would otherwise take weeks to perform at considerable cost, can now be ready for production within a day. This gives them more flexibility in taking an agile approach to constantly improve operations while minimizing cost and risk.",[11,3488,3489],{},"For instance, the automatic collection of all accounting information from connected hotels was streamlined, including timely alerts and automated reminders if necessary. Automation saves staff time and effort, allowing them to focus on providing better customer service and improving the overall guest experience.",[11,3491,3492],{},"Felix Kraemer, Head of Data & Analytics at H-Hotels.com:",[3494,3495,3496],"blockquote",{},[11,3497,3498],{},"\"The introduction of layline.io was a swift and painless process. The team at layline.io was incredibly responsive and supportive throughout the entire implementation process, which only took a few days. It was easy to set up and integrate with our existing systems.\"",[11,3500,3501],{},"He furthermore notes:",[3494,3503,3504],{},[11,3505,3506],{},"\"layline.io is a very cost-efficient solution for our business, as it has allowed us to streamline our operations, reduce costs associated with manual labor, and increase revenue through better decision-making. The promise of being completely self-sufficient was kept at 100%. The ROI of the software is already evident a few weeks into production. We are extremely satisfied with the results, and we are just getting started to fully utilize the capabilities.\"",[26,3508,3273],{"id":3272},[11,3510,3511],{},"Data integration software can provide significant benefits to many industries looking to optimize their operations and stay competitive in a rapidly evolving market. H-Hotels.com continues to leverage layline.io to gain valuable insights into its operations, automate their processes, and personalize the guests' experiences, resulting in increased revenue and customer satisfaction. If you're looking to transform your business, layline.io provides many possibilities to help you achieve your goals.",[359,3513,3514,3519,3526],{},[362,3515,3087,3516,1515],{},[226,3517,3092],{"href":3090,"rel":3518},[2187],[362,3520,3521,3522,1515],{},"Read more about H-Hotels.com ",[226,3523,3092],{"href":3524,"rel":3525},"https://h-hotels.com/",[2187],[362,3527,3109,3528,1515],{},[226,3529,3113],{"href":3112},{"title":255,"searchDepth":256,"depth":256,"links":3531},[3532],{"id":3272,"depth":256,"text":3273},"2023-05-21","The hospitality industry has seen significant technological advancements in recent years. This includes data integration software being one of the most critical tools that the industry can leverage to optimize their operations.","/images/blog/2023-05-21/H-Hotels_Logo-hor_Color_SC-U.svg",{},"/blog/2023-05-21-h-hotels-digital-transformation",{"title":3466,"description":3534},{"loc":3537},"blog/2023-05-21-h-hotels-digital-transformation","DboHwcNIpahp0nLgaVoVYsTG_5_Disb_VqlEvpv9p5I",{"id":3543,"title":3544,"body":3545,"category":269,"date":3638,"description":3639,"extension":271,"featured":2934,"image":3640,"meta":3641,"navigation":272,"path":3642,"readTime":3643,"seo":3644,"sitemap":3645,"stem":3646,"__hash__":3647},"blog/blog/2023-05-03-event-based-processing.md","The Advantage of Event-Based Asynchronous Data Processing",{"type":8,"value":3546,"toc":3627},[3547,3551,3554,3557,3560,3563,3566,3569,3573,3577,3580,3584,3587,3591,3594,3598,3601,3605,3608,3610,3613,3616],[26,3548,3550],{"id":3549},"unlocking-the-potential-of-asynchronous-data-processing","Unlocking the Potential of Asynchronous Data Processing",[11,3552,3553],{},"Asynchronous data processing has revolutionized the way businesses process large volumes of data. It is a technique in which multiple tasks can be executed simultaneously without waiting for the completion of one task to start another. This approach has resulted in improved efficiency and productivity, making it a popular choice among businesses of all sizes.",[11,3555,3556],{},"One of the most significant advantages is that it allows businesses to handle large volumes of data in a shorter time frame. This is particularly important if you are dealing with massive amounts of data, such as e-commerce websites, financial institutions, communication carriers, and healthcare providers. With asynchronous data processing, these businesses can process data faster, which results in quicker turnaround times and improved customer satisfaction.",[11,3558,3559],{},"Event-based processing takes this a step further. It involves the execution of tasks based on events or triggers. These events can be anything from a customer placing an order to a system error being detected. When an event occurs, the relevant task is initiated, and other tasks can continue to run in parallel. This approach ensures that data processing occurs in real-time and reduces the time taken to complete tasks.",[11,3561,3562],{},"Another example is in the field of IoT (Internet of Things). IoT devices generate a large amount of data, and event-based processing allows for real-time processing of this data. For instance, consider a smart home system that controls the lighting, temperature, and security of a house. The system can be designed to respond to specific events, such as a change in temperature or motion detected by a security camera. Based on these events, the system can initiate tasks such as adjusting the temperature or sending an alert to the homeowner's phone.",[11,3564,3565],{},"The combination of event-based and asynchronous processing allows businesses to handle spikes in data volume without affecting performance. For example, during peak periods, such as Black Friday or Cyber Monday, e-commerce websites experience a significant increase in traffic and sales. Increased volume can be handled without impacting the performance.",[11,3567,3568],{},"It also enables businesses to scale their operations quickly and efficiently. With traditional synchronous processing, businesses would need to invest in expensive hardware and software to handle increased data volume. However, with asynchronous processing, businesses can add more processing power as needed, without the need for significant investments in hardware or software.",[26,3570,3572],{"id":3571},"key-benefits-of-event-based-asynchronous-data-processing","Key Benefits of Event-based Asynchronous Data Processing",[339,3574,3576],{"id":3575},"quick-processing-times","Quick Processing Times",[11,3578,3579],{},"One of the primary benefits is the ability to quickly process large amounts of data. By breaking data processing tasks down into smaller, more manageable events, event-based processing can help organizations process data more quickly and efficiently than traditional batch processing methods. This can be especially useful in situations where real-time data processing is required, such as in financial trading or online advertising.",[339,3581,3583],{"id":3582},"reduced-risk-of-errors","Reduced Risk of Errors",[11,3585,3586],{},"Another key benefit is the ability to reduce the risk of errors. By breaking data processing tasks down into smaller events, event-based processing helps to identify and address errors more quickly and efficiently than traditional batch processing methods. This can help to reduce the risk of errors and improve the accuracy of data processing operations.",[339,3588,3590],{"id":3589},"prioritization-of-critical-tasks","Prioritization of Critical Tasks",[11,3592,3593],{},"It also offers the ability to prioritize critical tasks. By breaking data processing tasks down into smaller events, event-based processing can help organizations prioritize critical tasks and ensure that they are processed in a timely and efficient manner. This can be especially useful in situations where certain tasks are more time-sensitive than others, such as in emergency response or healthcare.",[339,3595,3597],{"id":3596},"scalability-of-operations","Scalability of Operations",[11,3599,3600],{},"Scaling is becoming increasingly important. Event-based asynchronous data processing helps to scale operations more easily and efficiently than traditional batch processing methods. This can be especially useful in situations where data processing needs are constantly changing, which is pretty much the case everywhere.",[339,3602,3604],{"id":3603},"elimination-of-system-crashes","Elimination of System Crashes",[11,3606,3607],{},"Finally, it can help organizations eliminate system crashes. It ensures that data processing operations run smoothly and efficiently.",[26,3609,3273],{"id":3272},[11,3611,3612],{},"Businesses of all sizes can benefit from event-based asynchronous data processing. From small startups to large-scale enterprises, this technology can improve efficiency, reduce costs, and increase agility. By automating tasks based on events, businesses can achieve faster processing times, reduce manual intervention, and eliminate the risk of errors.",[11,3614,3615],{},"As you may have guessed, layline.io is built on an event-based asynchronous processing architecture :-) and a supporter of the reactive manifesto. Check out the myriad of use-cases which layline.io supports and learn more about how it works.",[359,3617,3618,3623],{},[362,3619,3087,3620,1515],{},[226,3621,3092],{"href":3090,"rel":3622},[2187],[362,3624,3109,3625,1515],{},[226,3626,3113],{"href":3112},{"title":255,"searchDepth":256,"depth":256,"links":3628},[3629,3630,3637],{"id":3549,"depth":256,"text":3550},{"id":3571,"depth":256,"text":3572,"children":3631},[3632,3633,3634,3635,3636],{"id":3575,"depth":540,"text":3576},{"id":3582,"depth":540,"text":3583},{"id":3589,"depth":540,"text":3590},{"id":3596,"depth":540,"text":3597},{"id":3603,"depth":540,"text":3604},{"id":3272,"depth":256,"text":3273},"2023-05-03","In today's fast-paced world, every business is looking for better ways to process large volumes of data. Traditional data processing methods have their limitations, which is why companies are increasingly exploring event-based asynchronous data processing.","/images/blog/2023-05-03/event-based-processing.jpg",{},"/blog/2023-05-03-event-based-processing","4 min",{"title":3544,"description":3639},{"loc":3642},"blog/2023-05-03-event-based-processing","NvnVggjXPC9frd45xIUk2uyHfbYcSAyjNG19eWjd79o",{"id":3649,"title":3650,"body":3651,"category":269,"date":4100,"description":4101,"extension":271,"featured":2934,"image":3658,"meta":4102,"navigation":272,"path":4103,"readTime":1340,"seo":4104,"sitemap":4105,"stem":4106,"__hash__":4107},"blog/blog/2022-11-04-kafka-comparison.md","layline.io - A Kafka Comparison",{"type":8,"value":3652,"toc":4088},[3653,3659,3663,3666,3670,3679,3701,3705,3708,3728,3731,3734,3737,3740,3744,3747,3758,3761,3765,3768,3771,3794,3797,3817,3820,3826,3829,3833,3836,3839,3843,3846,3849,3853,3856,3859,3862,3867,3870,3872,3875,3878,3882,3885,4077],[11,3654,3655],{},[92,3656],{"alt":3657,"src":3658},"layline.io loves Kafka","/images/blog/2022-11-04/laylineio_loves_kafka_twitter.png",[26,3660,3662],{"id":3661},"how-does-laylineio-compare-to-kafka","How does layline.io compare to Kafka?",[11,3664,3665],{},"This is a question which we hear from time to time. We wonder why. To better understand, let's look at what Kafka is:",[26,3667,3669],{"id":3668},"what-is-kafka","What is Kafka?",[11,3671,3672,3673,3678],{},"This is how ",[226,3674,3677],{"href":3675,"rel":3676},"https://aws.amazon.com/msk/what-is-kafka/",[2187],"AWS"," describes it:",[3494,3680,3681,3684,3687,3698],{},[11,3682,3683],{},"\"Apache Kafka is a distributed data store optimized for ingesting and processing streaming data in real-time. Streaming data is data that is continuously generated by thousands of data sources, which typically send the data records in simultaneously. A streaming platform needs to handle this constant influx of data, and process the data sequentially and incrementally.",[11,3685,3686],{},"Kafka provides three main functions to its users:",[359,3688,3689,3692,3695],{},[362,3690,3691],{},"Publish and subscribe to streams of records.",[362,3693,3694],{},"Effectively store streams of records in the order in which records were generated.",[362,3696,3697],{},"Process streams of records in real time.",[11,3699,3700],{},"Kafka is primarily used to build real-time streaming data pipelines and applications that adapt to the data streams. It combines messaging, storage, and stream processing to allow storage and analysis of both historical and real-time data.\"",[26,3702,3704],{"id":3703},"our-take-on-kafka","Our take on Kafka",[11,3706,3707],{},"There is a bit of technobabble in the above description which we need to address. It talks about streaming processing, real-time etc. What does this all mean in the context of Kafka?",[604,3709,3710,3716,3722],{},[362,3711,3712,3715],{},[246,3713,3714],{},"Kafka is a data storage solution first."," It should be viewed as a special type of database in which data is stored in queues (topics). There are various ways on how these queues can be written to (publish) and then read from (subscribe). Queues follow the FIFO principle (first-in-first-out). There is an argument that Kafka is not a store but a streaming event processor, but this is misleading in our view. Kafka is designed to store data and then for Consumers to read that data quickly. It is designed to get rid of stored data after a pre-configured retention period regardless of whether the data was consumed, or not. In that sense it is a temporary data store with a few very special, albeit useful, features.",[362,3717,3718,3721],{},[246,3719,3720],{},"Individual Processes can publish to queues (Producers) or subscribe to queues (Consumers)."," If none of the out-of-the-box connectors (see below) suffices as a producer/consumer for your purpose (likely), then you need to custom code one yourself (most use Java, but there are other options).",[362,3723,3724,3727],{},[246,3725,3726],{},"Kafka can run in a cloud native, distributed environment",", providing resilience and scalability.",[11,3729,3730],{},"Confluent (the company) has added some more features to Kafka such as pre-made \"Connectors\". Connectors are special types of Producers and Consumers which can read/write special types of data sources/sinks from/to Kafka topics. They are quite limited and specialized in what they can do.",[11,3732,3733],{},"In addition to this they created the capability to \"filter, route, aggregate, and merge\" data using \"ksql\". It suggests that you can filter and route information in real-time from Kafka topics. Sounds great. It is, however, just another type of Consumer which reads data from a Kafka topic, then filter, aggregate, merge and route results to another topic for some other Consumer to read. The best way to compare this logically, is to use the analogy of a table-driven database (e.g. Oracle) in which you copy data from one table to another, using SQL; except with Kafka it is a lot more complicated.",[11,3735,3736],{},"Kafka has some, but no meaningful capabilities to transform data. One of the big obstacles here is that Kafka has no inherent capability to parse data, and thus be able to work on it. It only supports very few limited data formats. Anything out of the ordinary (likely) requires custom coding Producers and Consumers to do the job. This again is like any other database which cares mostly about internals, not externals. Overall, it is fair to say, that Kafka does not actually process data. It merely stores data. Any other scenario implies chaining up atomic topics with Producers and Consumers.",[11,3738,3739],{},"Kafka is also known to be rather hard to operate. There is no comprehensive user interface. Pretty everything is configured in config files and operated from the command line.",[26,3741,3743],{"id":3742},"summary","Summary",[11,3745,3746],{},"The main use of Kafka is for:",[359,3748,3749,3752,3755],{},[362,3750,3751],{},"Fast data storage",[362,3753,3754],{},"For large volumes",[362,3756,3757],{},"Which are both produced and consumed quickly",[11,3759,3760],{},"We love Kafka for this purpose. It is great, and we frequently make use of it in implementations, even though there are several other solutions to achieve this as well.",[26,3762,3764],{"id":3763},"what-is-laylineio","What is layline.io?",[11,3766,3767],{},"layline.io.io is a fast, scalable, resilient event data processor. It can ingest, process and output data in real-time. Processing means \"doing\" something with the data, as opposed to what Kafka does (store). A major difference to Kafka for example is that in layline.io everything revolves around the notion of \"Workflows\". Workflows reflect data-driven logic which commonly resemble complex data orchestration.",[11,3769,3770],{},"This translates to:",[359,3772,3773,3776,3779,3782,3785,3788,3791],{},[362,3774,3775],{},"interpreting the data,",[362,3777,3778],{},"analyzing it,",[362,3780,3781],{},"deciding and potentially enriching it by consulting other sources in real-time",[362,3783,3784],{},"create stats",[362,3786,3787],{},"filter it,",[362,3789,3790],{},"route it,",[362,3792,3793],{},"integrating otherwise disparate data sources and sink.",[11,3795,3796],{},"and do all of this in:",[359,3798,3799,3802,3805,3808,3811,3814],{},[362,3800,3801],{},"real-time,",[362,3803,3804],{},"transactionally secure (option),",[362,3806,3807],{},"without storage overhead,",[362,3809,3810],{},"configurable,",[362,3812,3813],{},"UI-driven",[362,3815,3816],{},"and much more",[11,3818,3819],{},"Example Workflow:",[11,3821,3822],{},[92,3823],{"alt":3824,"src":3825},"Complex Workflow in layline.io -> Try this with Kafka!","/images/blog/2022-11-04/1a0e6f5d.png",[11,3827,3828],{},"Kafka does not support Workflows by design. Anything which may be interpreted as a Workflow in Kafka is rather an attempt to \"sell\" a chaining of Kafka queues and Consumers / Producers as a Workflow. Note, however, that each of these are individual entities which are not aware of each other. There is no overarching (transaction) control, nor is there actual support for Workflows within Kafka.",[26,3830,3832],{"id":3831},"combining-kafka-and-laylineio","Combining Kafka and layline.io",[11,3834,3835],{},"From the viewpoint of Kafka, this means that layline.io is seen as a Producer (write) or Consumer (read). From the viewpoint of layline.io, Kafka is seen as an event data store, comparable to other data stores like SQL DBs, NOSQL DBs, or even the file system. That's a great combination, depending on the use case. layline.io in this context acts as the data orchestration element between a theoretical unlimited number of Kafka topics, and other sources and sinks outside the Kafka sphere.",[11,3837,3838],{},"From that angle, Kafka and layline.io are extremely complementary, not competitive. The overlap is minimal. We do not see a meaningful scenario where a potential customer would decide between one or the other, but rather for one and the other.",[26,3840,3842],{"id":3841},"how-do-kafka-users-cope-today","How do Kafka users cope today?",[11,3844,3845],{},"Today's typical Kafka user uses Kafka as what it is: A special kind of event data store. To write and read data to/from it. He mostly custom codes consumers and producers. These custom coded parts then must contain points 1 through 6 from above (hint: they will not). They furthermore do not warrant resilience, scalability, reporting, monitoring and everything else which would have to be expected from such components. They are often built using simple scripting tools like Python up to using more sophisticated microservice frameworks like Spring Boot et al.",[11,3847,3848],{},"Instead of this, they could just be using layline.io and get all the above.",[26,3850,3852],{"id":3851},"real-deployment-example","Real Deployment Example",[11,3854,3855],{},"This actual customer uses layline.io to take care of processing all customer usage data (communication meta data).",[11,3857,3858],{},"Before the implementation of layline.io, they were in a situation which very much resembled the scenario described in the previous paragraph. Several Kafka queues were fed by singular custom coded processes. Other such processes read from those queues and wrote data to other targets in other formats and with some logic applied. A messy and error-prone architecture, which was costly to maintain, and almost impossible to manage.",[11,3860,3861],{},"After the layline.io implementation, in which the previous business logic and processors was replaced, the architecture looked as follows:",[11,3863,3864],{},[92,3865],{"alt":3852,"src":3866},"/images/blog/2022-11-04/ef52522c.png",[11,3868,3869],{},"The overall picture is considered the \"Solution\". It is important to understand, that layline.io is recognized as the actual solution, whereas Kafka is just another (important) store. Kafka was there before layline.io. It plays a comparatively small role (red arrow and box) in the overall solution, however. It serves as an intermittent data store, which is exactly what it is by definition. All intelligent data analysis, enrichment, filtering, transformation, routing, complex business logic, and much more is handled by layline.io. A task at a level which would be impossible to accomplish using Kafka. Asking what part of this is owed to Kafka, the customer's answer would likely be: \"5% of the overall solution\".",[26,3871,3273],{"id":3272},[11,3873,3874],{},"Kafka is widely considered a message bus, but it is really about data-at-rest facilitating other applications to put data-in-motion. In this context actual application from the viewpoint of a user is something which is fed with data from Kafka using a client API. layline.io on the contrary forms an integral part of your application's logic, if it is not the application itself (see example). You can imagine layline.io like the circulatory system plus logic, whereas Kafka is just an external well-organized reservoir. The overlap between Kafka and layline.io is therefore minimal.",[11,3876,3877],{},"A customer who is running Kafka, will not question whether layline.io could make sense \"in addition to Kafka\". Likewise, we would not question the use of Kafka as a data store, only whether this would be the right data store for the purpose. Customers would rather question how they solve issues (which Kafka does not address) using layline.io. They may be doing something new, or replace existing processes (e.g. microservices) which were custom coded to some extent in the past.",[26,3879,3881],{"id":3880},"appendix-quick-comparison-laylineio-kafka","Appendix: Quick Comparison layline.io \u003C> Kafka",[11,3883,3884],{},"Not a full comparison, but it helps:",[1958,3886,3887,3899],{},[1961,3888,3889],{},[1964,3890,3891,3894,3897],{},[1967,3892,3893],{},"Aspect",[1967,3895,3896],{},"Kafka",[1967,3898,252],{},[1977,3900,3901,3912,3923,3934,3945,3956,3967,3978,3989,3999,4008,4017,4028,4037,4047,4056,4067],{},[1964,3902,3903,3906,3909],{},[1982,3904,3905],{},"Type",[1982,3907,3908],{},"Message Queue",[1982,3910,3911],{},"Concurrency Platform",[1964,3913,3914,3917,3920],{},[1982,3915,3916],{},"Workflow support",[1982,3918,3919],{},"Not really. Just a store.",[1982,3921,3922],{},"Inherent part of the solution",[1964,3924,3925,3928,3931],{},[1982,3926,3927],{},"Data store",[1982,3929,3930],{},"Yes",[1982,3932,3933],{},"No",[1964,3935,3936,3939,3942],{},[1982,3937,3938],{},"Data format support",[1982,3940,3941],{},"No understanding of data formats out-of-the-box. Only in context with ksql some limited support for formats such as CSV, JSON, Avro, ProtoBuf.",[1982,3943,3944],{},"Complete understanding of data content. Strongly typed. Support for extremely complex data formats, such as ASCII and binary, hierarchical structures, ASN.1 etc.",[1964,3946,3947,3950,3953],{},[1982,3948,3949],{},"Business Logic",[1982,3951,3952],{},"No support",[1982,3954,3955],{},"Full support. This is a major difference between a store and a data processing solution.",[1964,3957,3958,3961,3964],{},[1982,3959,3960],{},"Data enrichment",[1982,3962,3963],{},"Not supported. No 3rd parties can be consulted for data enrichment.",[1982,3965,3966],{},"Full support.",[1964,3968,3969,3972,3975],{},[1982,3970,3971],{},"Real-time",[1982,3973,3974],{},"Kafka is a store. This can only be as real-time as whatever reads the data from the store (buffer).",[1982,3976,3977],{},"Full. As real-time as it gets. No intermittent storage. Data is processed and output instantly.",[1964,3979,3980,3983,3986],{},[1982,3981,3982],{},"Custom metrics",[1982,3984,3985],{},"No custom metrics specific to your use case",[1982,3987,3988],{},"Any type of custom metric (e.g. \"4711 customers have signed up for service y in last time interval\")",[1964,3990,3991,3994,3997],{},[1982,3992,3993],{},"Performance",[1982,3995,3996],{},"High",[1982,3998,3996],{},[1964,4000,4001,4004,4006],{},[1982,4002,4003],{},"Scalable",[1982,4005,3930],{},[1982,4007,3930],{},[1964,4009,4010,4013,4015],{},[1982,4011,4012],{},"Resilient / HA",[1982,4014,3930],{},[1982,4016,3930],{},[1964,4018,4019,4022,4025],{},[1982,4020,4021],{},"Persistent",[1982,4023,4024],{},"Yes. That's the purpose of Kafka.",[1982,4026,4027],{},"No. Not the purpose of layline.io, but works great with persistence layers, such as Kafka.",[1964,4029,4030,4033,4035],{},[1982,4031,4032],{},"UI-driven configuration",[1982,4034,3933],{},[1982,4036,3930],{},[1964,4038,4039,4042,4044],{},[1982,4040,4041],{},"Memory intensity",[1982,4043,3996],{},[1982,4045,4046],{},"Low",[1964,4048,4049,4052,4054],{},[1982,4050,4051],{},"Hardware footprint",[1982,4053,3996],{},[1982,4055,4046],{},[1964,4057,4058,4061,4064],{},[1982,4059,4060],{},"Open Source",[1982,4062,4063],{},"Yes, for community edition. No for confluent solution (e.g. ksql)",[1982,4065,4066],{},"Not yet.",[1964,4068,4069,4072,4075],{},[1982,4070,4071],{},"Ready cloud offer",[1982,4073,4074],{},"Yes, for confluent",[1982,4076,4066],{},[359,4078,4079,4084],{},[362,4080,3087,4081,1515],{},[226,4082,3092],{"href":3090,"rel":4083},[2187],[362,4085,3109,4086,1515],{},[226,4087,3113],{"href":3112},{"title":255,"searchDepth":256,"depth":256,"links":4089},[4090,4091,4092,4093,4094,4095,4096,4097,4098,4099],{"id":3661,"depth":256,"text":3662},{"id":3668,"depth":256,"text":3669},{"id":3703,"depth":256,"text":3704},{"id":3742,"depth":256,"text":3743},{"id":3763,"depth":256,"text":3764},{"id":3831,"depth":256,"text":3832},{"id":3841,"depth":256,"text":3842},{"id":3851,"depth":256,"text":3852},{"id":3272,"depth":256,"text":3273},{"id":3880,"depth":256,"text":3881},"2022-11-04","How does layline.io compare to Kafka? This is a question which we hear from time to time. We wonder why.",{},"/blog/2022-11-04-kafka-comparison",{"title":3650,"description":4101},{"loc":4103},"blog/2022-11-04-kafka-comparison","i3qbm8sI19D0DL022OpqvmAe63CjVZpEjwt9gGhKLqQ",{"id":4109,"title":4110,"body":4111,"category":3329,"date":4584,"description":4585,"extension":271,"featured":2934,"image":4586,"meta":4587,"navigation":272,"path":4588,"readTime":553,"seo":4589,"sitemap":4590,"stem":4591,"__hash__":4592},"blog/blog/2022-10-01-http-client-requests.md","Sample Series: Http-Client-Requests",{"type":8,"value":4112,"toc":4565},[4113,4117,4120,4124,4127,4131,4134,4138,4147,4155,4159,4162,4168,4173,4176,4182,4186,4189,4195,4198,4202,4205,4211,4215,4218,4221,4227,4230,4233,4237,4240,4245,4248,4254,4257,4263,4279,4286,4289,4293,4296,4302,4305,4310,4313,4319,4323,4326,4332,4335,4338,4341,4344,4350,4353,4357,4360,4366,4370,4373,4379,4383,4386,4394,4400,4408,4412,4415,4421,4425,4428,4431,4437,4440,4442,4445,4482,4494,4498,4520,4554],[26,4114,4116],{"id":4115},"what-we-will-show","What we will show",[11,4118,4119],{},"In this article we will showcase how to quickly configure a layline.io workflow to query a REST-API using the Yahoo Finance API as an example. We will create a workflow which polls the Yahoo for stock quotes in frequent intervals.",[26,4121,4123],{"id":4122},"setting-up-the-http-service","Setting up the Http-Service",[11,4125,4126],{},"First we fire up the web-UI and create a new Project \"sample-http-client-yahoo\". We then create a Http-Service Asset \"Yahoo-Finance-Source\".",[339,4128,4130],{"id":4129},"what-are-services","What are Services?",[11,4132,4133],{},"Services in layline.io represent specific assets which encapsulate complex interfaces definitions and then expose them to other Assets as simple named function calls for easy invocation. REST-API definitions fit that pattern, as well as for example database statements. For this purpose layline.io provides a number of Service Assets which are geared towards the specific interface types, such as Http, JDBC, Aerospike, and more. In our example, we will use the Http-Service Asset to access the Yahoo Finance API.",[339,4135,4137],{"id":4136},"the-yahoo-stock-quotes-rest-api","The Yahoo stock quotes REST-API",[11,4139,4140,4141,4146],{},"Yahoo Finance provides a very comprehensive API to access financial information publicly traded companies. For this demo we are going through ",[226,4142,4145],{"href":4143,"rel":4144},"https://rapidapi.com/",[2187],"Rapid-API"," which acts as a proxy service. If you don't know Rapid-API you should check it out. It's really cool.",[11,4148,4149,4150,4154],{},"You can look up the exact URL for the service, and it's description ",[226,4151,3092],{"href":4152,"rel":4153},"https://rapidapi.com/apidojo/api/yh-finance",[2187],", then check for the path \"market/v2/get-quotes\".",[339,4156,4158],{"id":4157},"configuring-the-http-service","Configuring the Http-Service",[11,4160,4161],{},"Using the Configuration Center we create a Http-Service Asset:",[11,4163,4164],{},[92,4165],{"alt":4166,"src":4167},"Creating the Http-Service Asset","/images/blog/2022-10-01/cdf2aa79.png",[4169,4170,4172],"h4",{"id":4171},"host","Host",[11,4174,4175],{},"Next we define the generic host address at which the data can be accessed. This does not require credentials, but API Keys which we will define later.",[11,4177,4178],{},[92,4179],{"alt":4180,"src":4181},"Host address","/images/blog/2022-10-01/21cba603.png",[4169,4183,4185],{"id":4184},"requests","Requests",[11,4187,4188],{},"Before we define Functions, we make sure that layline.io understand the requests and corresponding responses. You can define any number of requests. Here we only define \"GetQuotes\" which should retrieve the stock quotes and related data for a given number of stock ticker symbols. This is also where the path to the request is defined. Additionally, we configure the parameters required to invoke the API (region and symbols, as well as the RapidAPI key and host). We provide the values when we define the Source Asset in a next step.",[11,4190,4191],{},[92,4192],{"alt":4193,"src":4194},"Defining the \"GetQuotes\" request","/images/blog/2022-10-01/07210dbd.png",[11,4196,4197],{},"If we wanted to use other requests, we could just simply add them here.",[4169,4199,4201],{"id":4200},"response","Response",[11,4203,4204],{},"Next we define possible responses to our request. We expect a JSON document which we have defined in \"Quotes\". layline.io is strongly typed. In our case we simply map the response to the internal \"AnyMap\" type. Additional responses could be 3xx to 5xx failures which we are also interested in.",[11,4206,4207],{},[92,4208],{"alt":4209,"src":4210},"Defining expected responses","/images/blog/2022-10-01/c121ee0d.png",[4169,4212,4214],{"id":4213},"functions","Functions",[11,4216,4217],{},"Now that we have defined request and responses, we can create the abstract functions which we want to expose internally.",[11,4219,4220],{},"We create a function by the name of \"GetQuotes\". It references the request by the same name which we have created above, and assign possible responses to this request (\"Quotes\").",[11,4222,4223],{},[92,4224],{"alt":4225,"src":4226},"Configuring the Function","/images/blog/2022-10-01/f4a337fb.png",[11,4228,4229],{},"We can now access the Yahoo Finance API by simply invoking the function \"GetQuotes\" and pass the required parameters to it. We will see an example for this when we define the Service Source below.",[11,4231,4232],{},"Alas, quite a bit to configure for a first Http-Service. But note that additional requests to the same host can be added very quickly.",[26,4234,4236],{"id":4235},"configuring-the-service-source","Configuring the Service-Source",[11,4238,4239],{},"We plan to constantly query the API for the latest stock data and will want to treat it as a regular data source in the workflow. We need to define a Service-Source Asset for this. A Service-Source Asset is a special type of Source that uses Functions defined in Service Assets, just like the one we have defined above.",[11,4241,4242],{},[92,4243],{"alt":4236,"src":4244},"/images/blog/2022-10-01/18df9a2e.png",[11,4246,4247],{},"We link the Http-Service we have defined to this Source:",[11,4249,4250],{},[92,4251],{"alt":4252,"src":4253},"Linking the Http-Service Asset to the Service-Source","/images/blog/2022-10-01/a7e6c6dc.png",[11,4255,4256],{},"Finally, we set up the API polling frequency and the actual API call parameters.",[11,4258,4259],{},[92,4260],{"alt":4261,"src":4262},"Configuring Polling and Processing Parameters","/images/blog/2022-10-01/a44e8c61.png",[11,4264,4265,4266,4270,4271,4274,4275,4278],{},"Via a cron-table-like setting we define a query interval of 10 seconds. ",[4267,4268,4269],"code",{},"Object name"," denotes the name of the object which will be returned by each query. Note that we add the date and time to the object name to distinguish each response object. We use a constant parameter for polling, which means nothing else than that we want to pass hard coded parameters to the Service Function instead of dynamic parameters. So we are always invoking the function for the same stock ticker symbols in our example. The ",[4267,4272,4273],{},"Constant Type"," describes the path to the actual Service Function which we want to call and its request object. For now this needs to be entered manually, but will be user guided shortly. We enter ",[4267,4276,4277],{},"Yahoo.Finance.Functions.GetQuotes.Request"," here.",[11,4280,4281,4282,4285],{},"Lastly, we provide the parameters which we want to pass to the ",[4267,4283,4284],{},"GetQuotes"," function in JSON-format. Remember that we have defined them when we created the Http-Service Asset.",[11,4287,4288],{},"That's it, we can now create the actual Workflow.",[26,4290,4292],{"id":4291},"creating-the-workflow","Creating the Workflow",[11,4294,4295],{},"We head over to the Workflow UI and set up a simple Workflow with a Stream-Input and a JavaScript Processor.",[11,4297,4298],{},[92,4299],{"alt":4300,"src":4301},"Workflow we are creating","/images/blog/2022-10-01/14d4e5da.png",[11,4303,4304],{},"In the settings for the Service-Input Asset we link it to the Http-Source which we have just created.",[11,4306,4307],{},[92,4308],{"alt":4252,"src":4309},"/images/blog/2022-10-01/8e54b1e6.png",[11,4311,4312],{},"For the Javascript Processor we add a very simple script which outputs the received message to the stream log. This is how we can inspect the results via the Audit Trail in the UI. Of course, you would normally not output the results to the log, but process it downstream in the Workflow.",[11,4314,4315],{},[92,4316],{"alt":4317,"src":4318},"Javascript output to stream log","/images/blog/2022-10-01/2573c4b1.png",[26,4320,4322],{"id":4321},"deploy-to-cluster-and-run","Deploy to Cluster and run",[11,4324,4325],{},"We are ready to test the Workflow. To do so we need to deploy it to a Reactive Engine Cluster. You can simply use the one on your laptop (single node). Or if you have a larger layline.io cluster elsewhere you, can deploy it there. To deploy we switch to the DEPLOYMENT tab of the Project:",[11,4327,4328],{},[92,4329],{"alt":4330,"src":4331},"Deployment to local cluster setup","/images/blog/2022-10-01/ff30d3ca.png",[11,4333,4334],{},"We create an Engine Configuration to deploy the Project. This defines the parts of the Project which we wish to deploy. In our example that would be the one Workflow we defined, as well as the Environment Asset that goes along with it.",[11,4336,4337],{},"Since we want to deploy to the local cluster we pick \"Deploy to Cluster\" and then our pre-defined \"Local Cluster\" setup. Every deployment needs a Tag. We use \"yahoo-quotes-\" followed by a macro \"${build-timestamp}\" to identify the Deployment. The macro will be replaced by a timestamp upon deployment transfer. This ensures that we always get a different tag with each Deployment.",[11,4339,4340],{},"We select to deploy all Workflows which we have created (just one). Lastly we start the transfer of the deployment by clicking \"TRANSFER DEPLOYMENT TO CLUSTER\" (Make sure the Cluster you are deploying to is up and running).",[11,4342,4343],{},"If the deployment went well you should see this:",[11,4345,4346],{},[92,4347],{"alt":4348,"src":4349},"Deployment result","/images/blog/2022-10-01/6d79c68d.png",[11,4351,4352],{},"Otherwise, an error message will be displayed, guiding you on how to fix the problem.",[339,4354,4356],{"id":4355},"activating-the-deployment","Activating the Deployment",[11,4358,4359],{},"We should now be ready to activate the Deployment. For this, we switch to the \"CLUSTER\" tab. This gives us a view of all things \"cluster\". In case you are managing more than one cluster, make sure you have the correct one selected from the top left drop-down-box.",[11,4361,4362],{},[92,4363],{"alt":4364,"src":4365},"Activating a Deployment setup","/images/blog/2022-10-01/ec4f64d9.png",[339,4367,4369],{"id":4368},"make-it-the-default-deployment","Make it the default Deployment",[11,4371,4372],{},"Select \"Deployment Storage\" from the tree on the left. This is where we find all the Deployments which are currently known to the selected cluster. In our example screenshot, in section \"Deployment Configuration\" we see 1 \"DeploymentRoot\": This is the basic empty default deployment which is always present. 2 \"yahoo-quotes-20221011093550\": This is the Deployment we just transferred to the cluster. To now activate the new Deployment on the cluster select it and then check the box \"is the cluster's default deployment\".",[11,4374,4375],{},[92,4376],{"alt":4377,"src":4378},"Make a Deployment the default","/images/blog/2022-10-01/ed0604a0.png",[339,4380,4382],{"id":4381},"schedule","Schedule",[11,4384,4385],{},"Now that the Deployment is active on the Cluster, we need to check whether it is actually running any instances of the Workflow. If this is the first time you deployed this Workflow the answer is likely \"no\". Let's check:",[359,4387,4388,4391],{},[362,4389,4390],{},"Select the \"Cluster\" tab",[362,4392,4393],{},"Select the \"Scheduler Master\" entry in the tree on the left",[11,4395,4396],{},[92,4397],{"alt":4398,"src":4399},"Scheduling a Workflow","/images/blog/2022-10-01/be541084.png",[359,4401,4402,4405],{},[362,4403,4404],{},"In the Scheduler Settings Box select the Scheduler node in the tree",[362,4406,4407],{},"Next make sure that the target number of instances is set to at least 1. Setting it to a higher number will start more instances of the same Workflow. Within a few seconds you should see the instance started. You only need to do this once. Next time you deploy the Workflow, the Reactive Engine will remember the number of instances you want to run off of this Workflow. You can also define the Scheduler settings you want as part of a Deployment. That's for another showcase, though.",[339,4409,4411],{"id":4410},"engine-status","Engine Status",[11,4413,4414],{},"Switch over to the \"Engine\" tab. Make sure that all Engine categories are green. Also check the status of individual Assets in the tree to the left.",[11,4416,4417],{},[92,4418],{"alt":4419,"src":4420},"Engine status with Deployment activated","/images/blog/2022-10-01/429a2db7.png",[339,4422,4424],{"id":4423},"checking-results","Checking Results",[11,4426,4427],{},"As soon as you have scheduled the Workflow (above), it starts polling Yahoo Finance interface for results every few seconds.",[11,4429,4430],{},"Head over to the Audit Trail tab to see the results show up in the Stream Log in JSON-format:",[11,4432,4433],{},[92,4434],{"alt":4435,"src":4436},"Viewing result in Audit Trail","/images/blog/2022-10-01/bf23d064.png",[11,4438,4439],{},"Of course, this output is for demo purposes only. In a real working environment you would process this data downstream in the Workflow.",[26,4441,3743],{"id":3742},[11,4443,4444],{},"This showcase highlights how you can create a Http-Client-Workflow on-the-fly without much hassle. And you get a lot more with that out-of-the-box:",[359,4446,4447,4453,4459,4464,4470,4476],{},[362,4448,4449,4452],{},[246,4450,4451],{},"Reactive"," — Embraces the reactive processing paradigm. Fully built on reactive stream management at the core",[362,4454,4455,4458],{},[246,4456,4457],{},"High scalability"," — Scales within one engine instance and beyond across multiple engines and distributed nodes",[362,4460,4461,4463],{},[246,4462,2977],{}," — Failover safe in distributed environments. Full 24/7 operation and upgradability",[362,4465,4466,4469],{},[246,4467,4468],{},"Automatic deployment"," — Deploy changed configurations with one click",[362,4471,4472,4475],{},[246,4473,4474],{},"Real-time and batch"," — Run both real-time and batch data integrations using the same platform",[362,4477,4478,4481],{},[246,4479,4480],{},"Metrics"," — Automatic metric generation to use in your favorite monitoring and alerting toolkit (e.g. Prometheus)",[11,4483,4484,4485,4490,4491,4493],{},"There are too many features to explain here. For more information please check the ",[226,4486,4489],{"href":4487,"rel":4488},"https://doc.layline.io/",[2187],"documentation"," or simply contact us at ",[226,4492,3113],{"href":3112},". Thanks for reading!",[26,4495,4497],{"id":4496},"resources","Resources",[1958,4499,4500,4510],{},[1961,4501,4502],{},[1964,4503,4504,4507],{},[1967,4505,4506],{},"#",[1967,4508,4509],{},"Resource",[1977,4511,4512],{},[1964,4513,4514,4517],{},[1982,4515,4516],{},"1",[1982,4518,4519],{},"Github: Sample Http-Client for Yahoo Finance (input test files in the directory _test_files of the Project)",[1958,4521,4522,4531],{},[1961,4523,4524],{},[1964,4525,4526,4528],{},[1967,4527,4506],{},[1967,4529,4530],{},"Documentation",[1977,4532,4533,4543],{},[1964,4534,4535,4537],{},[1982,4536,4516],{},[1982,4538,4539],{},[226,4540,4542],{"href":4487,"rel":4541},[2187],"Getting Started",[1964,4544,4545,4548],{},[1982,4546,4547],{},"2",[1982,4549,4550],{},[226,4551,4553],{"href":4487,"rel":4552},[2187],"Generic Format Asset",[359,4555,4556,4561],{},[362,4557,3087,4558,1515],{},[226,4559,3092],{"href":3090,"rel":4560},[2187],[362,4562,3109,4563,1515],{},[226,4564,3113],{"href":3112},{"title":255,"searchDepth":256,"depth":256,"links":4566},[4567,4568,4573,4574,4575,4582,4583],{"id":4115,"depth":256,"text":4116},{"id":4122,"depth":256,"text":4123,"children":4569},[4570,4571,4572],{"id":4129,"depth":540,"text":4130},{"id":4136,"depth":540,"text":4137},{"id":4157,"depth":540,"text":4158},{"id":4235,"depth":256,"text":4236},{"id":4291,"depth":256,"text":4292},{"id":4321,"depth":256,"text":4322,"children":4576},[4577,4578,4579,4580,4581],{"id":4355,"depth":540,"text":4356},{"id":4368,"depth":540,"text":4369},{"id":4381,"depth":540,"text":4382},{"id":4410,"depth":540,"text":4411},{"id":4423,"depth":540,"text":4424},{"id":3742,"depth":256,"text":3743},{"id":4496,"depth":256,"text":4497},"2022-10-01","ReST interfaces are popular and abundant. We show you how to configure Http-Client requests within layline.io using Yahoo Finance as an example.","/images/blog/2022-10-01/3b3c4499.png",{},"/blog/2022-10-01-http-client-requests",{"title":4110,"description":4585},{"loc":4588},"blog/2022-10-01-http-client-requests","DfFK3NbccbJA_3z_RpVJZRjS1F7Gh2fK4YnIcoz-nyU",{"id":4594,"title":4595,"body":4596,"category":3329,"date":4777,"description":4778,"extension":271,"featured":2934,"image":4779,"meta":4780,"navigation":272,"path":4781,"readTime":1027,"seo":4782,"sitemap":4783,"stem":4784,"__hash__":4785},"blog/blog/2022-07-01-sniffing-realtime-data.md","Sniffing Real-Time Data",{"type":8,"value":4597,"toc":4764},[4598,4602,4605,4608,4611,4614,4618,4621,4624,4628,4631,4637,4640,4643,4648,4652,4655,4675,4678,4682,4685,4689,4692,4696,4699,4709,4712,4718,4721,4724,4728,4731,4734,4740,4743,4745,4748,4751,4753],[26,4599,4601],{"id":4600},"the-challenge-of-debugging-data-in-complex-processing-streams","The challenge of debugging data in complex processing streams",[11,4603,4604],{},"The world uses the notion of Workflows to define complex processing scenarios. One of the characteristics of such Workflows is, that they can be very complex in that they contain many steps (processes) which are interconnected based on various conditions.",[11,4606,4607],{},"Data traverses these Workflows, triggers actions and is pushed further along based on its content and state, among other things. The more complex a Workflow and the data traversing it, the harder it is to spot issues within the Workflow and the data, which may lead to undesired outcomes.",[11,4609,4610],{},"Pretty much all such systems, are black boxes in the way they process information. You can look at the input, see what's coming out the other side and try to understand what happened in between. You may also be able to add debug-output statements in the code to capture snapshots of data in order to get to the bottom of things, or outright dump all data to a log and then try to browse through it and connect the dots.",[11,4612,4613],{},"All of this, is just a major pain, frustrating and very time-consuming. There has to be a better way.",[26,4615,4617],{"id":4616},"sniffing-data-in-laylineio-workflows","Sniffing data in layline.io Workflows",[11,4619,4620],{},"Instead of making the Workflow spit out debug information, a smarter approach would be to treat the Workflow more like a computer network. Monitoring data in computer networks is non-invasive in that data is probed (\"sniffed\") while traversing the network. In other words: It travels freely, but is picked up at points of interest, for further analysis without disturbance of the flow.",[11,4622,4623],{},"layline.io works the same way. It supports probing individual workflows for data while they are fully running their tasks. Below we will explain how this works and take a look at the advantages.",[26,4625,4627],{"id":4626},"starting-a-sniffer-session","Starting a Sniffer Session",[11,4629,4630],{},"Probes are initiated and their results viewed through the web-based Configuration Center. Probes can be started on any running Workflow. To do so, head over to \"/Operations —> Audit Trail/\". You will see a tab which says \"Sniffer Sessions\" which is what we will use:",[11,4632,4633],{},[92,4634],{"alt":4635,"src":4636},"Sniffer Sessions View","/images/blog/2022-07-01/29068bb6.png",[11,4638,4639],{},"Sniffer Sessions - as the name implies - open a session which collects data and then closes it after a predefined volume and/or time. The results of the session are persisted, and you can take your time to then analyse the results.",[11,4641,4642],{},"To start a new sniffer session, you need to click on the little eye button to the right. This will bring up a dialog which helps you to define a new sniffer session:",[11,4644,4645],{},[92,4646],{"alt":4635,"src":4647},"/images/blog/2022-07-01/4a9811df.png",[339,4649,4651],{"id":4650},"pick-workflow","Pick Workflow",[11,4653,4654],{},"On the left we can pick a Workflow from all running Workflows. Choices are:",[359,4656,4657,4663,4669],{},[362,4658,4659,4662],{},[246,4660,4661],{},"Generally",": Probe Workflow regardless where it is running",[362,4664,4665,4668],{},[246,4666,4667],{},"Node level",": Probe Workflow running on specific node",[362,4670,4671,4674],{},[246,4672,4673],{},"Instance level",": Probe Workflow specific instance of Workflow running on specific node",[11,4676,4677],{},"This gives us granular control from where exactly we want to sniff data. Imagine running a specific Workflow on multiple nodes and instances, but only one of those instances seems to have issues. Here you can specify that you only want to sniff data from that particular instance instead of any Workflow instance.",[339,4679,4681],{"id":4680},"parameters","Parameters",[11,4683,4684],{},"On the right we have the Session Parameters. We can add parameters like an individual name for the session, set the maximum number of messages we want to probe for, as well as set a time limit for how long we want to listen for data in seconds.",[339,4686,4688],{"id":4687},"trigger","Trigger",[11,4690,4691],{},"Last not least we can define when sniffing shall be started. You can either instantly start sniffing any message that flies by, or only start whenever a new stream starts, or from the first instance which responds, etc.",[26,4693,4695],{"id":4694},"sniffer-execution","Sniffer Execution",[11,4697,4698],{},"Once you hit \"OK\" on the dialog, sniffing will commence. An open sniffer session will be created, waiting for data. Depending on your previous settings, the session will remain open until either the desired number of messages has been probed or the max-duration threshold has been reached.",[11,4700,4701,4705],{},[92,4702],{"alt":4703,"src":4704},"Sniffer Sessions Active","/images/blog/2022-07-01/798595bc.png",[92,4706],{"alt":4707,"src":4708},"Sniffer Sessions Closed","/images/blog/2022-07-01/75ffb707.png",[11,4710,4711],{},"As messages roll in, their meta-data are listed on the right-hand side of the Sniffer window with the most recent on the top. This contains information like the time, type of messages, where it was sniffed (instance), the name of the stream, and very importantly where in the workflow it was probed (location).",[11,4713,4714],{},[92,4715],{"alt":4716,"src":4717},"Sniffer Message Results","/images/blog/2022-07-01/869ea305.png",[11,4719,4720],{},"You may ask yourself from where in the Workflow messages are taken. As we can imagine, message content is prone to being modified as it traverses the Workflow's processors. So where is the best place to probe for data? At the beginning, then end, somewhere in the middle? The answer is, on every connecting \"line\" in a Workflow. This is again analogous to network sniffing, where you don't sniff in computers, but rather \"on the wire\" between computers.",[11,4722,4723],{},"layline.io likewise sniffs the same message on all connecting wires of a Workflow, which allows you to follow a message from beginning to end as it passes through the Workflow.",[26,4725,4727],{"id":4726},"analysis","Analysis",[11,4729,4730],{},"When selecting a message from the result list, its position from where it was sniffed will be displayed in the Workflow graph at the bottom section of the window. Just check for the orange dotted line. Selecting the same message in other locations on the list will show it in other locations in the Workflow graph. This allows you to actually check a message in all of its stages or processing and compare potential changes to the same message.",[11,4732,4733],{},"Last not least we would like to know what's inside the message. For this purpose click the \"+\" sign at the beginning of the message to unfold the message structure and see all data contained.",[11,4735,4736],{},[92,4737],{"alt":4738,"src":4739},"Sniffer Message Result Detail","/images/blog/2022-07-01/c9438fc4.png",[11,4741,4742],{},"Notice that the probing data is persisted. This is really convenient in case you are running a number of probes on the same Workflow and want to compare results between multiple sniffer sessions.",[26,4744,3743],{"id":3742},[11,4746,4747],{},"A usually cumbersome task of debugging and probing data from millions of messages is made super simple with layline.io. Like in a production line for coal, you can simply grab some from the conveyor belt as it flies by, and analyse it without disturbance of the manufacturing process.",[11,4749,4750],{},"Pretty cool.",[26,4752,4497],{"id":4496},[359,4754,4755,4760],{},[362,4756,3087,4757,1515],{},[226,4758,3092],{"href":3090,"rel":4759},[2187],[362,4761,3109,4762,1515],{},[226,4763,3113],{"href":3112},{"title":255,"searchDepth":256,"depth":256,"links":4765},[4766,4767,4768,4773,4774,4775,4776],{"id":4600,"depth":256,"text":4601},{"id":4616,"depth":256,"text":4617},{"id":4626,"depth":256,"text":4627,"children":4769},[4770,4771,4772],{"id":4650,"depth":540,"text":4651},{"id":4680,"depth":540,"text":4681},{"id":4687,"depth":540,"text":4688},{"id":4694,"depth":256,"text":4695},{"id":4726,"depth":256,"text":4727},{"id":3742,"depth":256,"text":3743},{"id":4496,"depth":256,"text":4497},"2022-07-01","It's hard to understand what's actually happening in complex processing scenarios. layline.io helps by providing probing tools to get insights into inner workings of complex Workflows at runtime.","/images/blog/2022-07-01/47118bb6.png",{},"/blog/2022-07-01-sniffing-realtime-data",{"title":4595,"description":4778},{"loc":4781},"blog/2022-07-01-sniffing-realtime-data","XCixQNLDYP6wCILoC1-2ycnXYzq0SPOJ8QXQFrwF3Yc",{"id":4787,"title":4788,"body":4789,"category":3329,"date":5303,"description":5304,"extension":271,"featured":2934,"image":4841,"meta":5305,"navigation":272,"path":5306,"readTime":1027,"seo":5307,"sitemap":5308,"stem":5309,"__hash__":5310},"blog/blog/2022-05-11-asn1-format.md","ASN.1 Format Configuration",{"type":8,"value":4790,"toc":5288},[4791,4795,4798,4801,4810,4814,4817,4820,4824,4833,4836,4842,4846,4850,4853,5005,5009,5012,5017,5020,5026,5030,5033,5039,5042,5048,5057,5063,5067,5070,5073,5079,5082,5088,5091,5097,5101,5104,5110,5113,5124,5130,5133,5139,5142,5146,5149,5155,5161,5163,5166,5169,5172,5210,5212,5232,5273,5284],[26,4792,4794],{"id":4793},"intro-to-asn1","Intro to ASN.1",[11,4796,4797],{},"In our last article \"Damn that Data Format Hell\" we talked about how layline.io deals with the multitude of complex structural data formats by using a grammar language to describe almost any such format, and enabling you to instantly work with the data.",[11,4799,4800],{},"We also discussed, that this doesn't work for ALL formats you may be working with, but maybe 80% of all structured formats.",[11,4802,4803,4804,4809],{},"There are data exchange formats out there in the wild, however, which adhere to their own standard and come with their own grammar language. One of which is ",[226,4805,4808],{"href":4806,"rel":4807},"https://en.wikipedia.org/wiki/ASN.1",[2187],"Abstract Syntax Notation One",", aka ASN.1.",[26,4811,4813],{"id":4812},"laylineio-showcase","layline.io Showcase",[11,4815,4816],{},"In this article we will showcase how our users can define ASN.1 based formats using layline.io.",[11,4818,4819],{},"layline.io is an opinionated event data processor. It wraps the power of reactive stream management in a framework which makes setup, deployment and monitoring of large scale event data processing requirements easy and straightforward.",[26,4821,4823],{"id":4822},"td57-transferred-account-procedure-tap","TD.57 - Transferred Account Procedure (TAP)",[11,4825,4826,4827,4832],{},"One popular area where ASN.1 protocol format plays an important role is in ",[226,4828,4831],{"href":4829,"rel":4830},"https://www.gsma.com/aboutus/wp-content/uploads/2012/03/smsdataroamingexplained.pdf",[2187],"telecoms roaming"," and the ensuing TAP exchange process. TAP (Transferred Accounts Procedure) handles the financial settlement process between mobile networks for their roaming subscribers.",[11,4834,4835],{},"In a nutshell: If you use your cell phone in a visiting network (e.g. other country or simply other network), then the visited network needs to invoice your home network for the service provided. The information about who used their phone in a visited network, how long and who they called, etc. is all exchanged in TAP and RAP files.",[11,4837,4838],{},[92,4839],{"alt":4840,"src":4841},"Simplified TD.57 TAP procedure","/images/blog/2022-05-11/677fe80f.png",[26,4843,4845],{"id":4844},"our-example-tap311-file-format-configuration","Our example: TAP3.11 file format configuration",[339,4847,4849],{"id":4848},"tap311-grammar","TAP3.11 Grammar",[11,4851,4852],{},"This is what the beginning of the official TAP3.11 grammar looks like:",[4854,4855,4859],"pre",{"className":4856,"code":4857,"language":4858,"meta":255,"style":255},"language-asn1 shiki shiki-themes material-theme-lighter github-light github-dark","-- TD57_3.11.2\n--\n-- The following ASN.1 specification defines the abstract syntax for\n--        Data Record Format Version 03\n--                           Release 11\n...\n\nTAP-0311  DEFINITIONS IMPLICIT TAGS  ::=\n\nBEGIN\n\nDataInterChange ::= CHOICE\n{\n    transferBatch TransferBatch,\n    notification  Notification,\n...\n}\n...\nTransferBatch ::= [APPLICATION 1] SEQUENCE\n{\n    batchControlInfo       BatchControlInfo            OPTIONAL,\n    accountingInfo         AccountingInfo              OPTIONAL,\n    networkInfo            NetworkInfo                 OPTIONAL,\n    ...\n}\n","asn1",[4267,4860,4861,4869,4874,4879,4885,4891,4897,4903,4909,4914,4920,4925,4931,4937,4943,4949,4954,4960,4965,4971,4976,4982,4988,4994,5000],{"__ignoreMap":255},[4862,4863,4866],"span",{"class":4864,"line":4865},"line",1,[4862,4867,4868],{},"-- TD57_3.11.2\n",[4862,4870,4871],{"class":4864,"line":256},[4862,4872,4873],{},"--\n",[4862,4875,4876],{"class":4864,"line":540},[4862,4877,4878],{},"-- The following ASN.1 specification defines the abstract syntax for\n",[4862,4880,4882],{"class":4864,"line":4881},4,[4862,4883,4884],{},"--        Data Record Format Version 03\n",[4862,4886,4888],{"class":4864,"line":4887},5,[4862,4889,4890],{},"--                           Release 11\n",[4862,4892,4894],{"class":4864,"line":4893},6,[4862,4895,4896],{},"...\n",[4862,4898,4900],{"class":4864,"line":4899},7,[4862,4901,4902],{"emptyLinePlaceholder":272},"\n",[4862,4904,4906],{"class":4864,"line":4905},8,[4862,4907,4908],{},"TAP-0311  DEFINITIONS IMPLICIT TAGS  ::=\n",[4862,4910,4912],{"class":4864,"line":4911},9,[4862,4913,4902],{"emptyLinePlaceholder":272},[4862,4915,4917],{"class":4864,"line":4916},10,[4862,4918,4919],{},"BEGIN\n",[4862,4921,4923],{"class":4864,"line":4922},11,[4862,4924,4902],{"emptyLinePlaceholder":272},[4862,4926,4928],{"class":4864,"line":4927},12,[4862,4929,4930],{},"DataInterChange ::= CHOICE\n",[4862,4932,4934],{"class":4864,"line":4933},13,[4862,4935,4936],{},"{\n",[4862,4938,4940],{"class":4864,"line":4939},14,[4862,4941,4942],{},"    transferBatch TransferBatch,\n",[4862,4944,4946],{"class":4864,"line":4945},15,[4862,4947,4948],{},"    notification  Notification,\n",[4862,4950,4952],{"class":4864,"line":4951},16,[4862,4953,4896],{},[4862,4955,4957],{"class":4864,"line":4956},17,[4862,4958,4959],{},"}\n",[4862,4961,4963],{"class":4864,"line":4962},18,[4862,4964,4896],{},[4862,4966,4968],{"class":4864,"line":4967},19,[4862,4969,4970],{},"TransferBatch ::= [APPLICATION 1] SEQUENCE\n",[4862,4972,4974],{"class":4864,"line":4973},20,[4862,4975,4936],{},[4862,4977,4979],{"class":4864,"line":4978},21,[4862,4980,4981],{},"    batchControlInfo       BatchControlInfo            OPTIONAL,\n",[4862,4983,4985],{"class":4864,"line":4984},22,[4862,4986,4987],{},"    accountingInfo         AccountingInfo              OPTIONAL,\n",[4862,4989,4991],{"class":4864,"line":4990},23,[4862,4992,4993],{},"    networkInfo            NetworkInfo                 OPTIONAL,\n",[4862,4995,4997],{"class":4864,"line":4996},24,[4862,4998,4999],{},"    ...\n",[4862,5001,5003],{"class":4864,"line":5002},25,[4862,5004,4959],{},[339,5006,5008],{"id":5007},"create-asn1-asset","Create ASN.1 Asset",[11,5010,5011],{},"Let's set up TAP3.11 using layline.io. In the Project Asset View (1) pick Formats (2) and then ASN.1 Format (3).",[11,5013,5014],{},[92,5015],{"alt":5008,"src":5016},"/images/blog/2022-05-11/2ae0f54f.png",[11,5018,5019],{},"The Asset editor has two tabs \"Configuration\" and \"Modules\". On the \"Configuration\" tab (1) we name the Asset \"TAP3.11\" (2):",[11,5021,5022],{},[92,5023],{"alt":5024,"src":5025},"Naming the new Asset","/images/blog/2022-05-11/dd90ea59.png",[339,5027,5029],{"id":5028},"define-message-types","Define Message Types",[11,5031,5032],{},"TAP3.11 defines a number of different \"root\" message types. We are only going to identify three of them to work with in this example:",[11,5034,5035],{},[92,5036],{"alt":5037,"src":5038},"Available Message Types","/images/blog/2022-05-11/eb52fed3.png",[11,5040,5041],{},"In layline.io we map them to \"Header\", \"Detail\", and \"Trailer\" respectively:",[11,5043,5044],{},[92,5045],{"alt":5046,"src":5047},"Defining Message Types in layline.io","/images/blog/2022-05-11/08bef098.png",[11,5049,5050,5051,5056],{},"We are later going to debug output the data we read. Because some data is ",[226,5052,5055],{"href":5053,"rel":5054},"https://en.wikipedia.org/wiki/Binary-coded_decimal",[2187],"BCD-encoded"," and for readability's sake we use a built-in converter to decode them:",[11,5058,5059],{},[92,5060],{"alt":5061,"src":5062},"BCD En-/Decoding","/images/blog/2022-05-11/340d9386.png",[339,5064,5066],{"id":5065},"adding-the-tap311-grammar","Adding the TAP3.11 grammar",[11,5068,5069],{},"Switch to the Modules tab (2) of the ASN.1 Asset we have added (1).",[11,5071,5072],{},"An ASN.1 grammar can be made-up of multiple modules. In our case it is just one. We have added it and named it \"TAP.311-Module\" (3/4).",[11,5074,5075],{},[92,5076],{"alt":5077,"src":5078},"Adding ASN.1 grammar module","/images/blog/2022-05-11/92db0f17.png",[11,5080,5081],{},"You can find the original ASN.1 grammar for TAP3.11 in the written specification:",[11,5083,5084],{},[92,5085],{"alt":5086,"src":5087},"GSMA's TD.57 TAP3.11 ASN.1 Grammar Notation","/images/blog/2022-05-11/3dd12b0c.png",[11,5089,5090],{},"Let's copy and paste it to our ASN.1 Asset (1) (it's about 1,422 lines):",[11,5092,5093],{},[92,5094],{"alt":5095,"src":5096},"Pasting ASN.1 Grammar into layline.io","/images/blog/2022-05-11/cf8b1923.png",[339,5098,5100],{"id":5099},"interactive-testing-of-the-asn1-grammar","Interactive Testing of the ASN.1 Grammar",[11,5102,5103],{},"layline.io provides the ability to instantly check whether format setup matches the actual data. You can upload a sample file (1) directly into the Configuration Center (2) to perform a cross-check.",[11,5105,5106],{},[92,5107],{"alt":5108,"src":5109},"Uploading ASN.1 Sample File","/images/blog/2022-05-11/77a9d12e.png",[11,5111,5112],{},"layline.io will instantly:",[604,5114,5115,5118,5121],{},[362,5116,5117],{},"Generate a parser for the ASN.1 grammar",[362,5119,5120],{},"Parse the sample file",[362,5122,5123],{},"Feed the result back (3 & 4)",[11,5125,5126],{},[92,5127],{"alt":5128,"src":5129},"Viewing ASN.1 Sample File","/images/blog/2022-05-11/52a8d154.png",[11,5131,5132],{},"Switch to the \"Sample Messages\" tab (2) where we can get a more readable view on the file content:",[11,5134,5135],{},[92,5136],{"alt":5137,"src":5138},"Decoded ASN.1 Sample File Viewing","/images/blog/2022-05-11/63503960.png",[11,5140,5141],{},"Interactive testing means, that you now could go back to the grammar, change it, and then instantly see how this affects how this affects the interpretation of your sample data.",[26,5143,5145],{"id":5144},"sample-project","Sample Project",[11,5147,5148],{},"We put together a complete sample Project together with test-data which is ready to run in layline.io. The example simply reads and writes a TAP file and outputs its contents to the console in a JSON-format:",[11,5150,5151],{},[92,5152],{"alt":5153,"src":5154},"Sample Project TAP3 Workflow","/images/blog/2022-05-11/7b33ab31.png",[11,5156,5157],{},[92,5158],{"alt":5159,"src":5160},"Sample Project Debug Output","/images/blog/2022-05-11/a20b5543.png",[26,5162,3743],{"id":3742},[11,5164,5165],{},"Using layline.io you can not only define complex, distributed, scalable event-processing scenarios, but pretty much work with any data format. In this example we demonstrated how to deal with ASN.1 structured data.",[11,5167,5168],{},"Most other solutions will support specific formats only, as their parsers have been hard-coded to match a specific use-case. In layline.io there are literally millions of use-cases. That's why formats are never hard-coded, but configurable.",[11,5170,5171],{},"There are many popular ASN.1 protocols around the world of which we would like to list some below:",[359,5173,5174,5177,5180,5183,5186,5189,5192,5195,5198,5201,5204,5207],{},[362,5175,5176],{},"Ericsson AIR / AXE10 / CCN / CCR / MTAS / R3-R13 / SDP",[362,5178,5179],{},"Huawei IMS",[362,5181,5182],{},"4G UMTS / 5G IMS / 3GPP",[362,5184,5185],{},"TD.35 NRTRDE - Near Real Time Roaming Data Exchange",[362,5187,5188],{},"TD.57 / TAP2 / TAP3 Transferred Account Procedure",[362,5190,5191],{},"X.500 Directory Services",[362,5193,5194],{},"Lightweight Directory Access Protocol (LDAP)",[362,5196,5197],{},"PKCS Cryptography Standards",[362,5199,5200],{},"Simple Network Management Protocol (SNMP)",[362,5202,5203],{},"Signalling System No. 7 (SS7)",[362,5205,5206],{},"Long-Term Evolution (LTE)",[362,5208,5209],{},"And many more...",[26,5211,4497],{"id":4496},[1958,5213,5214,5223],{},[1961,5215,5216],{},[1964,5217,5218,5220],{},[1967,5219,4506],{},[1967,5221,5222],{},"Description",[1977,5224,5225],{},[1964,5226,5227,5229],{},[1982,5228,4516],{},[1982,5230,5231],{},"Github: Simple ASN.1 Project",[1958,5233,5234,5242],{},[1961,5235,5236],{},[1964,5237,5238,5240],{},[1967,5239,4506],{},[1967,5241,4530],{},[1977,5243,5244,5253,5262],{},[1964,5245,5246,5248],{},[1982,5247,4516],{},[1982,5249,5250],{},[226,5251,4542],{"href":4487,"rel":5252},[2187],[1964,5254,5255,5257],{},[1982,5256,4547],{},[1982,5258,5259],{},[226,5260,4553],{"href":4487,"rel":5261},[2187],[1964,5263,5264,5267],{},[1982,5265,5266],{},"3",[1982,5268,5269],{},[226,5270,5272],{"href":4806,"rel":5271},[2187],"ASN.1 on Wikipedia",[359,5274,5275,5280],{},[362,5276,3087,5277,1515],{},[226,5278,3092],{"href":3090,"rel":5279},[2187],[362,5281,3109,5282,1515],{},[226,5283,3113],{"href":3112},[5285,5286,5287],"style",{},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":255,"searchDepth":256,"depth":256,"links":5289},[5290,5291,5292,5293,5300,5301,5302],{"id":4793,"depth":256,"text":4794},{"id":4812,"depth":256,"text":4813},{"id":4822,"depth":256,"text":4823},{"id":4844,"depth":256,"text":4845,"children":5294},[5295,5296,5297,5298,5299],{"id":4848,"depth":540,"text":4849},{"id":5007,"depth":540,"text":5008},{"id":5028,"depth":540,"text":5029},{"id":5065,"depth":540,"text":5066},{"id":5099,"depth":540,"text":5100},{"id":5144,"depth":256,"text":5145},{"id":3742,"depth":256,"text":3743},{"id":4496,"depth":256,"text":4497},"2022-05-11","ASN.1 is still a popular data format. Learn how easy it is to configure any ASN.1 format in layline.io.",{},"/blog/2022-05-11-asn1-format",{"title":4788,"description":5304},{"loc":5306},"blog/2022-05-11-asn1-format","t2fWa8-AEBQaakD3YdasRRqoueZXvAE9MtEtpKp_UIc",{"id":5312,"title":5313,"body":5314,"category":3329,"date":6441,"description":6442,"extension":271,"featured":2934,"image":6281,"meta":6443,"navigation":272,"path":6444,"readTime":3127,"seo":6445,"sitemap":6446,"stem":6447,"__hash__":6448},"blog/blog/2022-05-02-data-format-hell.md","Damn that Data Format Hell",{"type":8,"value":5315,"toc":6420},[5316,5320,5323,5326,5329,5333,5336,5343,5347,5350,5367,5370,5378,5381,5384,5388,5395,5399,5402,5416,5420,5423,5426,5434,5437,6132,6139,6145,6151,6166,6172,6179,6185,6188,6194,6198,6201,6221,6225,6228,6242,6245,6249,6252,6266,6270,6276,6282,6285,6291,6297,6300,6304,6307,6312,6318,6323,6329,6333,6336,6362,6365,6367,6406,6417],[26,5317,5319],{"id":5318},"setting","Setting",[11,5321,5322],{},"You're the gal/guy who is tasked with taking care of and running the data processing infrastructure at ACME Corp. That's the complex and sensitive machinery that does all the heavy data lifting in your company, yet never reaps any rewards for it. It's the pump which may not stop pumping, it's the juice which keeps things going.",[11,5324,5325],{},"Yet when your parents, friends, even colleagues ask you what it is that you do at ACME, they're just dozing off in the middle of your enthusiastic explanation.",[11,5327,5328],{},"Let's face it: You don't get enough credit for what it is that you do for the world 🙃",[26,5330,5332],{"id":5331},"challenges","Challenges",[11,5334,5335],{},"The job you are doing doesn't require you to run around in a mechanic's outfit, but it sure feels like you have to constantly observe, mend, correct, and simply inject new mojo into that data crunching infrastructure.",[11,5337,5338,5339,5342],{},"While there are many day-to-day challenges, one of them is about ",[246,5340,5341],{},"CHANGE",". Yes, never change a running system, they say, but that's not how the world turns. One of the more frequent changes is …",[339,5344,5346],{"id":5345},"data-format-change","Data Format Change",[11,5348,5349],{},"AAHHHH! Red alert … sirens sounding …! When it comes to data format change you often see a number of challenges coming together at the same time:",[604,5351,5352,5355,5358,5361,5364],{},[362,5353,5354],{},"Data interfaces are often hard coded in some source code. You can't just \"change\" them (as the boss suggests), or at least it's not that simple.",[362,5356,5357],{},"During migration, data is received in both old and new format. Even with a small change, that's still two different formats which need to be processed in parallel.",[362,5359,5360],{},"It's not just about another field. Sometimes it's a whole added data structure and a bunch of other things all at the same time.",[362,5362,5363],{},"Changed data may require internal handling of additional information etc. Again, stuff may be hard coded and therefore code needs to change to accommodate.",[362,5365,5366],{},"You ran out of coffee. That's no help.",[11,5368,5369],{},"We have two major problems here which are:",[604,5371,5372,5375],{},[362,5373,5374],{},"Change of code and",[362,5376,5377],{},"handling of format migration.",[11,5379,5380],{},"This can turn out to be quite headache and require long cycles of planning, development, release, testing and finally deployment.",[11,5382,5383],{},"If there only was a better way to get this done quickly and in an easy way ...",[26,5385,5387],{"id":5386},"generic-data-formats-to-the-rescue","Generic Data Formats to the Rescue",[11,5389,5390,5391,5394],{},"layline.io features ",[246,5392,5393],{},"Generic Data Formats"," which can't solve all of the above challenges, but most of them, most of the time.",[339,5396,5398],{"id":5397},"what-are-generic-data-formats","What are Generic Data Formats?",[11,5400,5401],{},"As the name implies, this concept allows to define data formats in a generic fashion. To do so, it provides:",[604,5403,5404,5407,5410,5413],{},[362,5405,5406],{},"a language to define the structure (grammar) of the format you are trying to wrestle.",[362,5408,5409],{},"This language is making use of regular expressions to define and identify individual elements of a structure, and then",[362,5411,5412],{},"sub-elements of that structure, etc.",[362,5414,5415],{},"It is object-oriented in that you can define and reuse element-structures throughout.",[339,5417,5419],{"id":5418},"what-does-that-language-look-like","What does that language look like?",[11,5421,5422],{},"Let's have a look. For this purpose we will work with a super simple data format, which has comma separated, must have one Header record, 1..n detail records and a trailer records.",[11,5424,5425],{},"Example data of a simple Bank Transaction log:",[4854,5427,5432],{"className":5428,"code":5430,"language":5431},[5429],"language-text","H;Sample Bank Transactions\nD;20-Aug-2021;NEFT;23237.00;00.00;37243.31\nD;21-Aug-2021;NEFT;00.00;3724.33;33518.98\nT;100\n","text",[4267,5433,5430],{"__ignoreMap":255},[11,5435,5436],{},"And here is how this format defined within layline.io using the generic grammar language:",[4854,5438,5442],{"className":5439,"code":5440,"language":5441,"meta":255,"style":255},"language-javascript shiki shiki-themes material-theme-lighter github-light github-dark","format {\n  name = \"Bank Transactions\" \n  description = \"Random bank transactions\"\n\n  start-element = \"File\"\n  target-namespace = \"BankIn\"\n\n  elements = [\n    // File sequence\n    {\n      name = \"File\"\n      type = \"Sequence\"\n      references = [\n        { name = \"Header\", referenced-element = \"Header\" },\n        { name = \"Details\", max-occurs = \"unlimited\", referenced-element = \"Detail\" },\n        { name = \"Trailer\", referenced-element = \"Trailer\" }\n      ]\n    },\n    // Header record \n    {\n      name = \"Header\"\n      type = \"Separated\"\n      regular-expression = \"H\"\n      separator-regular-expression = \";\"\n      separator = \";\"\n      terminator-regular-expression = \"\\n\"\n      terminator = \"\\n\"\n      mapping = { message = \"Header\", element = \"BT_IN\" }\n      parts = [\n        { name = \"RECORD_TYPE\", type = \"RegExpr\", regular-expression = \"[^;\\n]*\", value.type = \"Text.String\" },\n        { name = \"FILENAME\", type = \"RegExpr\", regular-expression = \"[^;\\n]*\", value.type = \"Text.String\" }\n      ]\n    },\n    // Detail record\n    {\n      name = \"Detail\"\n      type = \"Separated\"\n      // ... similar structure\n    },\n    // Trailer record\n    {\n      name = \"Trailer\"\n      type = \"Separated\"\n      // ... similar structure\n    }\n  ]\n}\n","javascript",[4267,5443,5444,5453,5477,5492,5496,5516,5535,5539,5549,5555,5560,5573,5587,5596,5632,5681,5713,5718,5723,5728,5732,5744,5757,5774,5790,5803,5821,5835,5873,5883,5953,6015,6020,6025,6031,6036,6049,6062,6068,6073,6079,6084,6097,6110,6115,6121,6127],{"__ignoreMap":255},[4862,5445,5446,5450],{"class":4864,"line":4865},[4862,5447,5449],{"class":5448},"su5hD","format ",[4862,5451,4936],{"class":5452},"sP7_E",[4862,5454,5455,5458,5462,5466,5470,5473],{"class":4864,"line":256},[4862,5456,5457],{"class":5448},"  name",[4862,5459,5461],{"class":5460},"smGrS"," =",[4862,5463,5465],{"class":5464},"sjJ54"," \"",[4862,5467,5469],{"class":5468},"s_sjI","Bank Transactions",[4862,5471,5472],{"class":5464},"\"",[4862,5474,5476],{"class":5475},"skxfh"," \n",[4862,5478,5479,5482,5484,5486,5489],{"class":4864,"line":540},[4862,5480,5481],{"class":5448},"  description",[4862,5483,5461],{"class":5460},[4862,5485,5465],{"class":5464},[4862,5487,5488],{"class":5468},"Random bank transactions",[4862,5490,5491],{"class":5464},"\"\n",[4862,5493,5494],{"class":4864,"line":4881},[4862,5495,4902],{"emptyLinePlaceholder":272},[4862,5497,5498,5501,5504,5507,5509,5511,5514],{"class":4864,"line":4887},[4862,5499,5500],{"class":5448},"  start",[4862,5502,5503],{"class":5460},"-",[4862,5505,5506],{"class":5448},"element",[4862,5508,5461],{"class":5460},[4862,5510,5465],{"class":5464},[4862,5512,5513],{"class":5468},"File",[4862,5515,5491],{"class":5464},[4862,5517,5518,5521,5523,5526,5528,5530,5533],{"class":4864,"line":4893},[4862,5519,5520],{"class":5448},"  target",[4862,5522,5503],{"class":5460},[4862,5524,5525],{"class":5448},"namespace",[4862,5527,5461],{"class":5460},[4862,5529,5465],{"class":5464},[4862,5531,5532],{"class":5468},"BankIn",[4862,5534,5491],{"class":5464},[4862,5536,5537],{"class":4864,"line":4899},[4862,5538,4902],{"emptyLinePlaceholder":272},[4862,5540,5541,5544,5546],{"class":4864,"line":4905},[4862,5542,5543],{"class":5448},"  elements",[4862,5545,5461],{"class":5460},[4862,5547,5548],{"class":5475}," [\n",[4862,5550,5551],{"class":4864,"line":4911},[4862,5552,5554],{"class":5553},"sutJx","    // File sequence\n",[4862,5556,5557],{"class":4864,"line":4916},[4862,5558,5559],{"class":5452},"    {\n",[4862,5561,5562,5565,5567,5569,5571],{"class":4864,"line":4922},[4862,5563,5564],{"class":5448},"      name",[4862,5566,5461],{"class":5460},[4862,5568,5465],{"class":5464},[4862,5570,5513],{"class":5468},[4862,5572,5491],{"class":5464},[4862,5574,5575,5578,5580,5582,5585],{"class":4864,"line":4927},[4862,5576,5577],{"class":5448},"      type",[4862,5579,5461],{"class":5460},[4862,5581,5465],{"class":5464},[4862,5583,5584],{"class":5468},"Sequence",[4862,5586,5491],{"class":5464},[4862,5588,5589,5592,5594],{"class":4864,"line":4933},[4862,5590,5591],{"class":5448},"      references",[4862,5593,5461],{"class":5460},[4862,5595,5548],{"class":5475},[4862,5597,5598,5601,5604,5606,5608,5611,5613,5616,5619,5621,5623,5625,5627,5629],{"class":4864,"line":4939},[4862,5599,5600],{"class":5452},"        {",[4862,5602,5603],{"class":5448}," name",[4862,5605,5461],{"class":5460},[4862,5607,5465],{"class":5464},[4862,5609,5610],{"class":5468},"Header",[4862,5612,5472],{"class":5464},[4862,5614,5615],{"class":5452},",",[4862,5617,5618],{"class":5475}," referenced-",[4862,5620,5506],{"class":5448},[4862,5622,5461],{"class":5460},[4862,5624,5465],{"class":5464},[4862,5626,5610],{"class":5468},[4862,5628,5472],{"class":5464},[4862,5630,5631],{"class":5452}," },\n",[4862,5633,5634,5636,5638,5640,5642,5645,5647,5649,5652,5655,5657,5659,5662,5664,5666,5668,5670,5672,5674,5677,5679],{"class":4864,"line":4945},[4862,5635,5600],{"class":5452},[4862,5637,5603],{"class":5448},[4862,5639,5461],{"class":5460},[4862,5641,5465],{"class":5464},[4862,5643,5644],{"class":5468},"Details",[4862,5646,5472],{"class":5464},[4862,5648,5615],{"class":5452},[4862,5650,5651],{"class":5475}," max-",[4862,5653,5654],{"class":5448},"occurs",[4862,5656,5461],{"class":5460},[4862,5658,5465],{"class":5464},[4862,5660,5661],{"class":5468},"unlimited",[4862,5663,5472],{"class":5464},[4862,5665,5615],{"class":5452},[4862,5667,5618],{"class":5475},[4862,5669,5506],{"class":5448},[4862,5671,5461],{"class":5460},[4862,5673,5465],{"class":5464},[4862,5675,5676],{"class":5468},"Detail",[4862,5678,5472],{"class":5464},[4862,5680,5631],{"class":5452},[4862,5682,5683,5685,5687,5689,5691,5694,5696,5698,5700,5702,5704,5706,5708,5710],{"class":4864,"line":4951},[4862,5684,5600],{"class":5452},[4862,5686,5603],{"class":5448},[4862,5688,5461],{"class":5460},[4862,5690,5465],{"class":5464},[4862,5692,5693],{"class":5468},"Trailer",[4862,5695,5472],{"class":5464},[4862,5697,5615],{"class":5452},[4862,5699,5618],{"class":5475},[4862,5701,5506],{"class":5448},[4862,5703,5461],{"class":5460},[4862,5705,5465],{"class":5464},[4862,5707,5693],{"class":5468},[4862,5709,5472],{"class":5464},[4862,5711,5712],{"class":5452}," }\n",[4862,5714,5715],{"class":4864,"line":4956},[4862,5716,5717],{"class":5475},"      ]\n",[4862,5719,5720],{"class":4864,"line":4962},[4862,5721,5722],{"class":5452},"    },\n",[4862,5724,5725],{"class":4864,"line":4967},[4862,5726,5727],{"class":5553},"    // Header record \n",[4862,5729,5730],{"class":4864,"line":4973},[4862,5731,5559],{"class":5452},[4862,5733,5734,5736,5738,5740,5742],{"class":4864,"line":4978},[4862,5735,5564],{"class":5448},[4862,5737,5461],{"class":5460},[4862,5739,5465],{"class":5464},[4862,5741,5610],{"class":5468},[4862,5743,5491],{"class":5464},[4862,5745,5746,5748,5750,5752,5755],{"class":4864,"line":4984},[4862,5747,5577],{"class":5448},[4862,5749,5461],{"class":5460},[4862,5751,5465],{"class":5464},[4862,5753,5754],{"class":5468},"Separated",[4862,5756,5491],{"class":5464},[4862,5758,5759,5762,5765,5767,5769,5772],{"class":4864,"line":4990},[4862,5760,5761],{"class":5475},"      regular-",[4862,5763,5764],{"class":5448},"expression",[4862,5766,5461],{"class":5460},[4862,5768,5465],{"class":5464},[4862,5770,5771],{"class":5468},"H",[4862,5773,5491],{"class":5464},[4862,5775,5776,5779,5781,5783,5785,5788],{"class":4864,"line":4996},[4862,5777,5778],{"class":5475},"      separator-regular-",[4862,5780,5764],{"class":5448},[4862,5782,5461],{"class":5460},[4862,5784,5465],{"class":5464},[4862,5786,5787],{"class":5468},";",[4862,5789,5491],{"class":5464},[4862,5791,5792,5795,5797,5799,5801],{"class":4864,"line":5002},[4862,5793,5794],{"class":5448},"      separator",[4862,5796,5461],{"class":5460},[4862,5798,5465],{"class":5464},[4862,5800,5787],{"class":5468},[4862,5802,5491],{"class":5464},[4862,5804,5806,5809,5811,5813,5815,5819],{"class":4864,"line":5805},26,[4862,5807,5808],{"class":5475},"      terminator-regular-",[4862,5810,5764],{"class":5448},[4862,5812,5461],{"class":5460},[4862,5814,5465],{"class":5464},[4862,5816,5818],{"class":5817},"s_hVV","\\n",[4862,5820,5491],{"class":5464},[4862,5822,5824,5827,5829,5831,5833],{"class":4864,"line":5823},27,[4862,5825,5826],{"class":5448},"      terminator",[4862,5828,5461],{"class":5460},[4862,5830,5465],{"class":5464},[4862,5832,5818],{"class":5817},[4862,5834,5491],{"class":5464},[4862,5836,5838,5841,5843,5846,5849,5851,5853,5855,5857,5859,5862,5864,5866,5869,5871],{"class":4864,"line":5837},28,[4862,5839,5840],{"class":5448},"      mapping",[4862,5842,5461],{"class":5460},[4862,5844,5845],{"class":5452}," {",[4862,5847,5848],{"class":5448}," message",[4862,5850,5461],{"class":5460},[4862,5852,5465],{"class":5464},[4862,5854,5610],{"class":5468},[4862,5856,5472],{"class":5464},[4862,5858,5615],{"class":5452},[4862,5860,5861],{"class":5448}," element",[4862,5863,5461],{"class":5460},[4862,5865,5465],{"class":5464},[4862,5867,5868],{"class":5468},"BT_IN",[4862,5870,5472],{"class":5464},[4862,5872,5712],{"class":5452},[4862,5874,5876,5879,5881],{"class":4864,"line":5875},29,[4862,5877,5878],{"class":5448},"      parts",[4862,5880,5461],{"class":5460},[4862,5882,5548],{"class":5475},[4862,5884,5886,5888,5890,5892,5894,5897,5899,5901,5904,5906,5908,5911,5913,5915,5918,5920,5922,5924,5927,5929,5932,5934,5936,5939,5942,5944,5946,5949,5951],{"class":4864,"line":5885},30,[4862,5887,5600],{"class":5452},[4862,5889,5603],{"class":5448},[4862,5891,5461],{"class":5460},[4862,5893,5465],{"class":5464},[4862,5895,5896],{"class":5468},"RECORD_TYPE",[4862,5898,5472],{"class":5464},[4862,5900,5615],{"class":5452},[4862,5902,5903],{"class":5448}," type",[4862,5905,5461],{"class":5460},[4862,5907,5465],{"class":5464},[4862,5909,5910],{"class":5468},"RegExpr",[4862,5912,5472],{"class":5464},[4862,5914,5615],{"class":5452},[4862,5916,5917],{"class":5475}," regular-",[4862,5919,5764],{"class":5448},[4862,5921,5461],{"class":5460},[4862,5923,5465],{"class":5464},[4862,5925,5926],{"class":5468},"[^;",[4862,5928,5818],{"class":5817},[4862,5930,5931],{"class":5468},"]*",[4862,5933,5472],{"class":5464},[4862,5935,5615],{"class":5452},[4862,5937,5938],{"class":5475}," value.",[4862,5940,5941],{"class":5448},"type",[4862,5943,5461],{"class":5460},[4862,5945,5465],{"class":5464},[4862,5947,5948],{"class":5468},"Text.String",[4862,5950,5472],{"class":5464},[4862,5952,5631],{"class":5452},[4862,5954,5956,5958,5960,5962,5964,5967,5969,5971,5973,5975,5977,5979,5981,5983,5985,5987,5989,5991,5993,5995,5997,5999,6001,6003,6005,6007,6009,6011,6013],{"class":4864,"line":5955},31,[4862,5957,5600],{"class":5452},[4862,5959,5603],{"class":5448},[4862,5961,5461],{"class":5460},[4862,5963,5465],{"class":5464},[4862,5965,5966],{"class":5468},"FILENAME",[4862,5968,5472],{"class":5464},[4862,5970,5615],{"class":5452},[4862,5972,5903],{"class":5448},[4862,5974,5461],{"class":5460},[4862,5976,5465],{"class":5464},[4862,5978,5910],{"class":5468},[4862,5980,5472],{"class":5464},[4862,5982,5615],{"class":5452},[4862,5984,5917],{"class":5475},[4862,5986,5764],{"class":5448},[4862,5988,5461],{"class":5460},[4862,5990,5465],{"class":5464},[4862,5992,5926],{"class":5468},[4862,5994,5818],{"class":5817},[4862,5996,5931],{"class":5468},[4862,5998,5472],{"class":5464},[4862,6000,5615],{"class":5452},[4862,6002,5938],{"class":5475},[4862,6004,5941],{"class":5448},[4862,6006,5461],{"class":5460},[4862,6008,5465],{"class":5464},[4862,6010,5948],{"class":5468},[4862,6012,5472],{"class":5464},[4862,6014,5712],{"class":5452},[4862,6016,6018],{"class":4864,"line":6017},32,[4862,6019,5717],{"class":5475},[4862,6021,6023],{"class":4864,"line":6022},33,[4862,6024,5722],{"class":5452},[4862,6026,6028],{"class":4864,"line":6027},34,[4862,6029,6030],{"class":5553},"    // Detail record\n",[4862,6032,6034],{"class":4864,"line":6033},35,[4862,6035,5559],{"class":5452},[4862,6037,6039,6041,6043,6045,6047],{"class":4864,"line":6038},36,[4862,6040,5564],{"class":5448},[4862,6042,5461],{"class":5460},[4862,6044,5465],{"class":5464},[4862,6046,5676],{"class":5468},[4862,6048,5491],{"class":5464},[4862,6050,6052,6054,6056,6058,6060],{"class":4864,"line":6051},37,[4862,6053,5577],{"class":5448},[4862,6055,5461],{"class":5460},[4862,6057,5465],{"class":5464},[4862,6059,5754],{"class":5468},[4862,6061,5491],{"class":5464},[4862,6063,6065],{"class":4864,"line":6064},38,[4862,6066,6067],{"class":5553},"      // ... similar structure\n",[4862,6069,6071],{"class":4864,"line":6070},39,[4862,6072,5722],{"class":5452},[4862,6074,6076],{"class":4864,"line":6075},40,[4862,6077,6078],{"class":5553},"    // Trailer record\n",[4862,6080,6082],{"class":4864,"line":6081},41,[4862,6083,5559],{"class":5452},[4862,6085,6087,6089,6091,6093,6095],{"class":4864,"line":6086},42,[4862,6088,5564],{"class":5448},[4862,6090,5461],{"class":5460},[4862,6092,5465],{"class":5464},[4862,6094,5693],{"class":5468},[4862,6096,5491],{"class":5464},[4862,6098,6100,6102,6104,6106,6108],{"class":4864,"line":6099},43,[4862,6101,5577],{"class":5448},[4862,6103,5461],{"class":5460},[4862,6105,5465],{"class":5464},[4862,6107,5754],{"class":5468},[4862,6109,5491],{"class":5464},[4862,6111,6113],{"class":4864,"line":6112},44,[4862,6114,6067],{"class":5553},[4862,6116,6118],{"class":4864,"line":6117},45,[4862,6119,6120],{"class":5452},"    }\n",[4862,6122,6124],{"class":4864,"line":6123},46,[4862,6125,6126],{"class":5475},"  ]\n",[4862,6128,6130],{"class":4864,"line":6129},47,[4862,6131,4959],{"class":5452},[339,6133,6135,6136,5861],{"id":6134},"the-format-element","The ",[4267,6137,6138],{},"format",[11,6140,6141,6142,6144],{},"Everything starts with the top-level ",[4267,6143,6138],{}," element:",[11,6146,6147],{},[92,6148],{"alt":6149,"src":6150},"Format element railroad diagram","/images/blog/2022-05-02/26f29e59.png",[11,6152,6153,6154,6157,6158,6161,6162,6165],{},"Besides a ",[4267,6155,6156],{},"name"," and ",[4267,6159,6160],{},"description",", it also has an array of ",[4267,6163,6164],{},"elements",". These elements define a number of sub-elements (classes), which can be of different types.",[339,6167,6169,6170],{"id":6168},"the-file-element-of-type-sequence","The File element of type ",[4267,6171,5584],{},[11,6173,6174,6175,6178],{},"This element defines a logical structure of sub-elements in its ",[4267,6176,6177],{},"references"," structure:",[11,6180,6181],{},[92,6182],{"alt":6183,"src":6184},"Sequence element railroad diagram","/images/blog/2022-05-02/2c8d5096.png",[11,6186,6187],{},"You may be able to tell that we are building a tree here:",[11,6189,6190],{},[92,6191],{"alt":6192,"src":6193},"Bank Transactions element tree","/images/blog/2022-05-02/b5198d49.png",[339,6195,6197],{"id":6196},"preliminary-conclusion","Preliminary conclusion",[11,6199,6200],{},"We have learned that the way the grammar language works, is as follows:",[604,6202,6203,6206,6209,6215,6218],{},[362,6204,6205],{},"A grammar consists of a number of elements",[362,6207,6208],{},"These elements come in different types which serve distinct purposes",[362,6210,6211,6212,6214],{},"The initial element is ",[4267,6213,6138],{}," which points to a starting element",[362,6216,6217],{},"You can define any number of additional elements",[362,6219,6220],{},"Some elements can then reference other elements",[339,6222,6224],{"id":6223},"how-about-other-more-complex-formats","How about other, more complex formats?",[11,6226,6227],{},"Here is what else works:",[359,6229,6230,6233,6236,6239],{},[362,6231,6232],{},"Very complex ASCII/Unicode formats",[362,6234,6235],{},"Conditional data parsing through managing conditional parser state",[362,6237,6238],{},"Binary structures",[362,6240,6241],{},"A mix of ASCII and binary formats",[11,6243,6244],{},"We believe this covers more than 80% of all data interchange formats.",[339,6246,6248],{"id":6247},"multiple-format-support","Multiple Format Support",[11,6250,6251],{},"You can define as many formats as you like. layline.io compiles all of them into a \"super-format\" at runtime. This allows you to:",[359,6253,6254,6257,6260,6263],{},[362,6255,6256],{},"reference all formats from anywhere",[362,6258,6259],{},"map data from one format to another",[362,6261,6262],{},"create new message instances based on a specific format",[362,6264,6265],{},"ingest or output data in any of the defined formats",[26,6267,6269],{"id":6268},"where-do-i-actually-configure-all-of-this","Where do I actually configure all of this?",[11,6271,6272,6273,2883],{},"We have provided a nice user interface to help you get all the configuration done. You can find it in the layline.io web-based Configuration Center under ",[4267,6274,6275],{},"Project --> Formats --> Generic Format",[11,6277,6278],{},[92,6279],{"alt":6280,"src":6281},"Generic Grammar Editor","/images/blog/2022-05-02/c9915c54.png",[11,6283,6284],{},"And it does not stop there. While you are defining your grammar, you can upload a sample data file, and see side-by-side whether your grammar matches the data file structure:",[11,6286,6287],{},[92,6288],{"alt":6289,"src":6290},"Grammar Sample File Viewer","/images/blog/2022-05-02/b8b634cd.png",[11,6292,6293],{},[92,6294],{"alt":6295,"src":6296},"Grammar Sample Messages Viewer","/images/blog/2022-05-02/8339768c.png",[11,6298,6299],{},"Pretty cool, eh?",[26,6301,6303],{"id":6302},"referencing-data-within-logic","Referencing Data within Logic",[11,6305,6306],{},"Once you have defined one or more grammars within layline.io, you can then access individual elements and structures within it:",[11,6308,6309],{},[246,6310,6311],{},"Example: Data Access within Mapping Asset",[11,6313,6314],{},[92,6315],{"alt":6316,"src":6317},"Data Access within Mapping Asset","/images/blog/2022-05-02/503566c3.png",[11,6319,6320],{},[246,6321,6322],{},"Example: Data Access within Javascript Asset",[11,6324,6325],{},[92,6326],{"alt":6327,"src":6328},"Data Access within Javascript Asset","/images/blog/2022-05-02/6ca44ccb.png",[26,6330,6332],{"id":6331},"dealing-with-format-changes","Dealing with Format Changes",[11,6334,6335],{},"Looking back at the challenges which we talked about at the beginning, it should now be clearer how easy it is to adapt to format changes.",[359,6337,6338,6341,6344,6359],{},[362,6339,6340],{},"Need another field? Just add it to the grammar.",[362,6342,6343],{},"Need another whole record structure? Again, just add it to the grammar.",[362,6345,6346,6347,6350,6351,6354,6355,6358],{},"Need to accommodate an old and a new version of a Detail record both at the same time? Easy: You can insert a ",[4267,6348,6349],{},"Choice"," element to allow for either a ",[4267,6352,6353],{},"Detail-Old"," or ",[4267,6356,6357],{},"Detail-New"," version.",[362,6360,6361],{},"Need to calculate content based on other content? Just add formulas.",[11,6363,6364],{},"It's all taken care of.",[26,6366,4497],{"id":4496},[1958,6368,6369,6377],{},[1961,6370,6371],{},[1964,6372,6373,6375],{},[1967,6374,4506],{},[1967,6376,5222],{},[1977,6378,6379,6389,6399],{},[1964,6380,6381,6383],{},[1982,6382,4516],{},[1982,6384,6385],{},[226,6386,6388],{"href":4487,"rel":6387},[2187],"Documentation: Getting Started",[1964,6390,6391,6393],{},[1982,6392,4547],{},[1982,6394,6395],{},[226,6396,6398],{"href":4487,"rel":6397},[2187],"Documentation: Generic Format Asset",[1964,6400,6401,6403],{},[1982,6402,5266],{},[1982,6404,6405],{},"Sample Project: Output to Kafka",[359,6407,6408,6413],{},[362,6409,3087,6410,1515],{},[226,6411,3092],{"href":3090,"rel":6412},[2187],[362,6414,3109,6415,1515],{},[226,6416,3113],{"href":3112},[5285,6418,6419],{},"html pre.shiki code .su5hD, html code.shiki .su5hD{--shiki-light:#90A4AE;--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .sP7_E, html code.shiki .sP7_E{--shiki-light:#39ADB5;--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .smGrS, html code.shiki .smGrS{--shiki-light:#39ADB5;--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .sjJ54, html code.shiki .sjJ54{--shiki-light:#39ADB5;--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .s_sjI, html code.shiki .s_sjI{--shiki-light:#91B859;--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .skxfh, html code.shiki .skxfh{--shiki-light:#E53935;--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .sutJx, html code.shiki .sutJx{--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#6A737D;--shiki-default-font-style:inherit;--shiki-dark:#6A737D;--shiki-dark-font-style:inherit}html pre.shiki code .s_hVV, html code.shiki .s_hVV{--shiki-light:#90A4AE;--shiki-default:#005CC5;--shiki-dark:#79B8FF}html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":255,"searchDepth":256,"depth":256,"links":6421},[6422,6423,6426,6437,6438,6439,6440],{"id":5318,"depth":256,"text":5319},{"id":5331,"depth":256,"text":5332,"children":6424},[6425],{"id":5345,"depth":540,"text":5346},{"id":5386,"depth":256,"text":5387,"children":6427},[6428,6429,6430,6432,6434,6435,6436],{"id":5397,"depth":540,"text":5398},{"id":5418,"depth":540,"text":5419},{"id":6134,"depth":540,"text":6431},"The format element",{"id":6168,"depth":540,"text":6433},"The File element of type Sequence",{"id":6196,"depth":540,"text":6197},{"id":6223,"depth":540,"text":6224},{"id":6247,"depth":540,"text":6248},{"id":6268,"depth":256,"text":6269},{"id":6302,"depth":256,"text":6303},{"id":6331,"depth":256,"text":6332},{"id":4496,"depth":256,"text":4497},"2022-05-02","Dealing with complex data formats and changes can be daunting. Learn how layline.io tackles this challenge using a configurable grammar language.",{},"/blog/2022-05-02-data-format-hell",{"title":5313,"description":6442},{"loc":6444},"blog/2022-05-02-data-format-hell","c7G5nEVzMDzc99Iu9lt6vKeb54LTM4iU7OoN3xxkvbo",{"id":6450,"title":6451,"body":6452,"category":3329,"date":6958,"description":6959,"extension":271,"featured":2934,"image":6960,"meta":6961,"navigation":272,"path":6962,"readTime":3127,"seo":6963,"sitemap":6964,"stem":6965,"__hash__":6966},"blog/blog/2022-03-21-output-to-kafka.md","Output to Kafka",{"type":8,"value":6453,"toc":6941},[6454,6458,6461,6470,6474,6478,6482,6485,6491,6511,6520,6524,6527,6533,6536,6543,6546,6557,6561,6568,6574,6591,6595,6598,6604,6608,6611,6617,6621,6624,6630,6634,6640,6646,6650,6653,6659,6662,6687,6691,6694,6704,6730,6734,6738,6741,6746,6753,6758,6760,6763,6768,6770,6775,6777,6782,6784,6789,6793,6796,6802,6805,6811,6813,6816,6847,6849,6889,6930],[26,6455,6457],{"id":6456},"what-were-demonstrating","What we're demonstrating",[11,6459,6460],{},"We're showcasing a simple layline.io Project which reads data from a file and outputs its contents into a Kafka topic.",[11,6462,6463,6464,6469],{},"To follow the showcase in a real setup, you can download the assets of this Project from the Resource section at the bottom. Read ",[226,6465,6468],{"href":6466,"rel":6467},"https://doc.layline.io/doc/wf-config/configuration.html#importing-a-project-directory",[2187],"this"," to learn how to import the project into your environment.",[26,6471,6473],{"id":6472},"configuration","Configuration",[339,6475,6477],{"id":6476},"the-workflow","The Workflow",[4169,6479,6481],{"id":6480},"outline","Outline",[11,6483,6484],{},"The workflow setup of this showcase was put together using layline.io's Workflow Editor and looks like this:",[11,6486,6487],{},[92,6488],{"alt":6489,"src":6490},"File-to-Kafka workflow","/images/blog/2022-03-21/eef4ae4b.png",[359,6492,6493,6499,6505],{},[362,6494,6495,6498],{},[246,6496,6497],{},"(1) Input Processor",": reading an input file with a header/detail/trailer structure, then",[362,6500,6501,6504],{},[246,6502,6503],{},"(2) Flow Processor",": mapping this into an output format, which is subsequently",[362,6506,6507,6510],{},[246,6508,6509],{},"(3) Output Processor",": written to a Kafka topic.",[11,6512,6513,6514,6519],{},"For the purpose of this showcase we are using a Kafka topic hosted by ",[226,6515,6518],{"href":6516,"rel":6517},"https://www.cloudkarafka.com/",[2187],"Cloud Karafka",". So if you run the showcase yourself, you do not require your own Kafka installation.",[4169,6521,6523],{"id":6522},"configuration-of-underlying-assets","Configuration of underlying Assets",[11,6525,6526],{},"The Workflow is based on a number of underlying Assets which are configured using the Asset Editor. The logical association between Workflow and Assets can be understood like this:",[11,6528,6529],{},[92,6530],{"alt":6531,"src":6532},"Logical association between Workflow and Assets","/images/blog/2022-03-21/9db0e16c.png",[11,6534,6535],{},"Workflows are comprised of a number of Processors which are connected by Links.",[11,6537,6538,6539,6542],{},"Processors are based on ",[246,6540,6541],{},"Assets",". Assets are configuration entities which are of a specific class and type. In the image above we can see a Processor by the name of \"InputFile\", which is of class Input Processor and type Stream Input Processor. It in turn relies on two other assets \"Source Asset\" and \"Format Asset\" which are of type File System Source and Generic Format respectively.",[11,6544,6545],{},"In short:",[359,6547,6548,6551,6554],{},[362,6549,6550],{},"A Workflow is composed of interconnected Processors",[362,6552,6553],{},"Processors rely on Assets which define them",[362,6555,6556],{},"Assets can rely on other Assets",[339,6558,6560],{"id":6559},"environment-asset-my-environment","Environment Asset: \"My-Environment\"",[11,6562,6563,6564,6567],{},"First: layline.io can help manage multiple different environments using ",[246,6565,6566],{},"Environment Assets",". This greatly helps when using the same Project in test-, staging-, and production environments which may require different directories, connections, passwords etc. We are using one Environment Asset (2) in this Project.",[11,6569,6570],{},[92,6571],{"alt":6572,"src":6573},"Environment Asset definition","/images/blog/2022-03-21/e4a8489b.png",[11,6575,6576,6577,6580,6581,6354,6584,6587,6588,1515],{},"Variables like these can be used throughout the project by using a macro like ",[4267,6578,6579],{},"${lay:dirIn}",". OS or Java system environment variables are prefixed with ",[4267,6582,6583],{},"env:",[4267,6585,6586],{},"sys:"," respectively, instead of ",[4267,6589,6590],{},"lay:",[339,6592,6594],{"id":6593},"stream-input-processor-inputfile","Stream Input Processor: \"InputFile\"",[11,6596,6597],{},"The Input Processor (name: InputFile / type: Stream Input Processor) takes care of reading the input files and forwarding the data downstream within the Workflow.",[11,6599,6600],{},[92,6601],{"alt":6602,"src":6603},"Stream Input Processor and Asset association","/images/blog/2022-03-21/fd2fd686.png",[4169,6605,6607],{"id":6606},"generic-format-asset-inputfileformat","Generic Format Asset: \"InputFileFormat\"",[11,6609,6610],{},"layline.io provides the means to define complex data structures with its own grammar language. The file in our example is a bank transaction sample. It has a header record with two fields, a number of detail records holding the transaction details, and finally a trailer record.",[11,6612,6613],{},[92,6614],{"alt":6615,"src":6616},"Generic Format Asset definition","/images/blog/2022-03-21/a1679036.png",[4169,6618,6620],{"id":6619},"file-system-source-asset-inputsource","File-System-Source Asset: \"InputSource\"",[11,6622,6623],{},"The \"InputSource\" is an Asset of type File System Source which is used to define where the file is read from.",[11,6625,6626],{},[92,6627],{"alt":6628,"src":6629},"File-System-Source Asset definition","/images/blog/2022-03-21/e3372610.png",[339,6631,6633],{"id":6632},"flow-processor-map","Flow Processor: Map",[11,6635,6135,6636,6639],{},[246,6637,6638],{},"Mapping Asset"," allows you to map values from the input to the output format.",[11,6641,6642],{},[92,6643],{"alt":6644,"src":6645},"Mapping Asset definition","/images/blog/2022-03-21/eb251cfc.png",[339,6647,6649],{"id":6648},"stream-output-processor-kafka","Stream Output Processor: Kafka",[11,6651,6652],{},"The last Processor in the Workflow is the Output Processor \"Kafka-Out\".",[11,6654,6655],{},[92,6656],{"alt":6657,"src":6658},"Stream Output Processor definition","/images/blog/2022-03-21/51e946c5.png",[11,6660,6661],{},"It depends on three underlying Assets:",[359,6663,6664,6670,6676,6681],{},[362,6665,6666,6669],{},[246,6667,6668],{},"Output Asset",": Defines Kafka topics and partitions we are writing to",[362,6671,6672,6675],{},[246,6673,6674],{},"Kafka Sink Asset",": The Sink that the Output Asset can use to send data to",[362,6677,6678,6680],{},[246,6679,4553],{},": Defines in what format to write the data to Kafka",[362,6682,6683,6686],{},[246,6684,6685],{},"Kafka Connection Asset",": Defines the physical Kafka connection parameters",[4169,6688,6690],{"id":6689},"kafka-connection-asset-cloud-karafka-connection","Kafka Connection Asset: \"Cloud-Karafka-Connection\"",[11,6692,6693],{},"To output to Kafka we first have to define a Kafka Connection Asset.",[11,6695,6696,6700],{},[92,6697],{"alt":6698,"src":6699},"Configuration of Kafka Connection","/images/blog/2022-03-21/8f8decca.png",[92,6701],{"alt":6702,"src":6703},"Kafka Settings","/images/blog/2022-03-21/e00b7507.png",[359,6705,6706,6712,6718,6724],{},[362,6707,6708,6711],{},[246,6709,6710],{},"(1) Bootstrap servers",": The addresses of one or more Bootstrap servers",[362,6713,6714,6717],{},[246,6715,6716],{},"(2) Use SSL",": Defines whether this is an SSL connection",[362,6719,6720,6723],{},[246,6721,6722],{},"(3) Authentication type",": SASL / Plaintext, or SASL / SCRAM",[362,6725,6726,6729],{},[246,6727,6728],{},"(4/5/6) Credentials",": Username/Password",[26,6731,6733],{"id":6732},"deploy-run","Deploy & Run",[339,6735,6737],{"id":6736},"transferring-the-deployment","Transferring the Deployment",[11,6739,6740],{},"To deploy we switch to the DEPLOYMENT tab of the Project:",[11,6742,6743],{},[92,6744],{"alt":4330,"src":6745},"/images/blog/2022-03-21/69990314.png",[11,6747,6748,6749,6752],{},"We create an ",[246,6750,6751],{},"Engine Configuration"," to deploy the Project. This defines the parts of the Project which we wish to deploy.",[11,6754,6755],{},[92,6756],{"alt":4348,"src":6757},"/images/blog/2022-03-21/8b91932b.png",[339,6759,4356],{"id":4355},[11,6761,6762],{},"We switch to the \"CLUSTER\" tab:",[11,6764,6765],{},[92,6766],{"alt":4364,"src":6767},"/images/blog/2022-03-21/960bceb0.png",[4169,6769,4369],{"id":4368},[11,6771,6772],{},[92,6773],{"alt":4377,"src":6774},"/images/blog/2022-03-21/9acd29ea.png",[4169,6776,4382],{"id":4381},[11,6778,6779],{},[92,6780],{"alt":4398,"src":6781},"/images/blog/2022-03-21/39a0f94a.png",[4169,6783,4411],{"id":4410},[11,6785,6786],{},[92,6787],{"alt":4419,"src":6788},"/images/blog/2022-03-21/466640a7.png",[339,6790,6792],{"id":6791},"feeding-the-test-file","Feeding the test file",[11,6794,6795],{},"To test, we feed our testfile to the input directory which we have configured.",[11,6797,6798],{},[92,6799],{"alt":6800,"src":6801},"Checking Audit Trail for successful processing status","/images/blog/2022-03-21/3002f0be.png",[11,6803,6804],{},"You can check the Cloud Karafka topic using a tool of your choice:",[11,6806,6807],{},[92,6808],{"alt":6809,"src":6810},"Kafka topic view (3rd party)","/images/blog/2022-03-21/d04498ff.png",[26,6812,3743],{"id":3742},[11,6814,6815],{},"This showcase highlights how you can create a File-to-Kafka Workflow on-the-fly without a hassle. And you get a lot more with that out-of-the-box:",[359,6817,6818,6823,6828,6833,6837,6842],{},[362,6819,6820,6822],{},[246,6821,4451],{}," — Embraces the reactive processing paradigm",[362,6824,6825,6827],{},[246,6826,4457],{}," — Scales within one engine instance and beyond",[362,6829,6830,6832],{},[246,6831,2977],{}," — Failover safe in distributed environments",[362,6834,6835,4469],{},[246,6836,4468],{},[362,6838,6839,6841],{},[246,6840,4474],{}," — Run both using the same platform",[362,6843,6844,6846],{},[246,6845,4480],{}," — Automatic metric generation for monitoring (e.g. Prometheus)",[26,6848,4497],{"id":4496},[1958,6850,6851,6859],{},[1961,6852,6853],{},[1964,6854,6855,6857],{},[1967,6856,4506],{},[1967,6858,5222],{},[1977,6860,6861,6868,6879],{},[1964,6862,6863,6865],{},[1982,6864,4516],{},[1982,6866,6867],{},"Github: Simple Kafka Project",[1964,6869,6870,6872],{},[1982,6871,4547],{},[1982,6873,6874,6875,6878],{},"input test files in the directory ",[4267,6876,6877],{},"_test_files"," of the Project",[1964,6880,6881,6883],{},[1982,6882,5266],{},[1982,6884,6885,6886],{},"Cloud Karafka credentials found in file ",[4267,6887,6888],{},"cloud-karafka-credentials.txt",[1958,6890,6891,6899],{},[1961,6892,6893],{},[1964,6894,6895,6897],{},[1967,6896,4506],{},[1967,6898,4530],{},[1977,6900,6901,6910,6920],{},[1964,6902,6903,6905],{},[1982,6904,4516],{},[1982,6906,6907],{},[226,6908,4542],{"href":4487,"rel":6909},[2187],[1964,6911,6912,6914],{},[1982,6913,4547],{},[1982,6915,6916],{},[226,6917,6919],{"href":4487,"rel":6918},[2187],"Importing a Project",[1964,6921,6922,6924],{},[1982,6923,5266],{},[1982,6925,6926],{},[226,6927,6929],{"href":4487,"rel":6928},[2187],"What are Assets, etc?",[359,6931,6932,6937],{},[362,6933,3087,6934,1515],{},[226,6935,3092],{"href":3090,"rel":6936},[2187],[362,6938,3109,6939,1515],{},[226,6940,3113],{"href":3112},{"title":255,"searchDepth":256,"depth":256,"links":6942},[6943,6944,6951,6956,6957],{"id":6456,"depth":256,"text":6457},{"id":6472,"depth":256,"text":6473,"children":6945},[6946,6947,6948,6949,6950],{"id":6476,"depth":540,"text":6477},{"id":6559,"depth":540,"text":6560},{"id":6593,"depth":540,"text":6594},{"id":6632,"depth":540,"text":6633},{"id":6648,"depth":540,"text":6649},{"id":6732,"depth":256,"text":6733,"children":6952},[6953,6954,6955],{"id":6736,"depth":540,"text":6737},{"id":4355,"depth":540,"text":4356},{"id":6791,"depth":540,"text":6792},{"id":3742,"depth":256,"text":3743},{"id":4496,"depth":256,"text":4497},"2022-03-21","Showcase on how to read data from a structured file, map record data, and output the data Kafka cloud.","/images/blog/2022-03-21/cabd8b86.png",{},"/blog/2022-03-21-output-to-kafka",{"title":6451,"description":6959},{"loc":6962},"blog/2022-03-21-output-to-kafka","ux97AjRYQocxBGa-KNMzm98HPq96LnQRNP1Pbn8ghWM",{"id":6968,"title":6969,"body":6970,"category":269,"date":7292,"description":7293,"extension":271,"featured":2934,"image":7166,"meta":7294,"navigation":272,"path":7295,"readTime":1340,"seo":7296,"sitemap":7297,"stem":7298,"__hash__":7299},"blog/blog/2022-03-01-kubernetes-workflows.md","Advantage of layline.io Workflows compared to traditional Microservices using the K8S/Docker model",{"type":8,"value":6971,"toc":7276},[6972,6975,6979,6983,6995,6998,7002,7008,7011,7015,7018,7021,7027,7030,7036,7039,7043,7046,7049,7053,7057,7063,7069,7076,7082,7089,7095,7098,7104,7107,7110,7116,7120,7123,7129,7132,7136,7139,7142,7145,7151,7155,7158,7161,7167,7170,7172,7231,7234,7237,7250,7252],[11,6973,6974],{},"The traditional Microservices model on Kubernetes/Docker has some disadvantages which result in overly complex management and resource consumption. In this article we explain how layline.io embraces container and container orchestration technology, while helping to resolve the aforementioned challenges with a better approach.",[26,6976,6978],{"id":6977},"quick-explainer-kubernetes-k8s-docker","Quick explainer: Kubernetes (K8S) & Docker",[339,6980,6982],{"id":6981},"containers","Containers",[11,6984,6985,6986,6988,6989,6994],{},"Programs running on Kubernetes are packaged into ",[246,6987,6982],{},". The advantage being that software and dependencies are packed together. One less thing to worry about and warranting the independence of the Container. There are tons of ready-made Containers downloadable from such portals as ",[226,6990,6993],{"href":6991,"rel":6992},"https://hub.docker.com/",[2187],"DockerHub",", packaging up all sorts software.",[11,6996,6997],{},"You can in theory pack many programs into one Container, but the recommendation and industry standard is one Container = one Process. This makes everything more granular, and you can replace individual containers (and therefore processes) more easily that way.",[339,6999,7001],{"id":7000},"pods","Pods",[11,7003,7004,7005,7007],{},"Containers don't run on their own, but are packaged in yet another \"container\". This time they're called ",[246,7006,7001],{},". Pods - among other things - manage virtual resources such as network, memory, CPU etc. for the Containers running within them.",[11,7009,7010],{},"It's important to understand, that you don't assign CPU power to a Container, but to a Pod. Therefore, if you run more than one Container in the same Pod, they all have to share the resources available to the Pod. For the same reasons you should not put more than one program in a Container, you should not put more than one Container in a Pod, unless multiple Containers are required to serve the purpose of the Microservice.",[26,7012,7014],{"id":7013},"resource-issues-when-scaling-in-kubernetes","Resource Issues when Scaling in Kubernetes",[11,7016,7017],{},"In Kubernetes the currency of scalability is Pods. To have more processing power, you fire up more Pods, also known as replication.",[11,7019,7020],{},"If you look at this design, a Pod itself is actually pretty static. If you want to adhere to utmost flexibility and be able to replace one program with another, then your Pod contains one Container, which in turn contains one program which will run as one process.",[11,7022,7023],{},[92,7024],{"alt":7025,"src":7026},"K8S/Docker packaging","/images/blog/2022-03-01/8dfbfcc8.png",[11,7028,7029],{},"Considering that the actual resources required for the container are configured on Pod-level, the image looks more like this:",[11,7031,7032],{},[92,7033],{"alt":7034,"src":7035},"K8S/Docker packaging w/ Resources","/images/blog/2022-03-01/161b5fd4.png",[11,7037,7038],{},"Check out the space marked as \"slack\". When you size the resources for a container, you have to account for some slack in CPU and memory. But because it's almost impossible to exactly determine the necessary resources for one program you end up having some reserve in every pod. If you run 100 of these, the slack adds up 100-fold. There is no resource-sharing between Pods. On top of this there is some overhead for each Pod and Node which gets added to the cluster. So, while overall the concept of K8S is great, it also adds considerable resource overhead overall.",[26,7040,7042],{"id":7041},"distribution-of-containers-within-a-kubernetes-cluster","Distribution of Containers within a Kubernetes Cluster",[11,7044,7045],{},"Pods are also the smallest denominator to distribute functionality within a Kubernetes Cluster. Let's say you have Microservices A, B and C and you want to distribute them unevenly within a Cluster, you either have individual Pods which each contain either A, B or C, or you have to have a number of Pods to make up all permutations of Pods containing containers A, B and C (e.g. a Pod with A and B, a Pod with A and C, etc.). That's a lot of Pods to manage and can quickly become overwhelming and inefficient.",[11,7047,7048],{},"Distribution of Pods is then a major configuration challenge within Kubernetes and/or your CI/CD tool of choice. Add to this the management of automated scaling and load balancing between Nodes and you end up having a major setup and monitoring headache.",[26,7050,7052],{"id":7051},"how-laylineio-deals-with-scalability-resources-and-distribution-in-a-kubernetes-cluster","How layline.io deals with scalability, resources, and distribution in a Kubernetes Cluster",[339,7054,7056],{"id":7055},"reactive-engine","Reactive Engine",[11,7058,7059,7060,7062],{},"layline.io introduces the ",[246,7061,7056],{},". The Engine serves as an execution context for Workflows which can be configured to run within a Reactive Engine. Workflows are comparable to Microservices in that they fulfil specific data processing tasks ranging from ingestion, analysis and enrichment as well as responding to query requests etc. Workflows are configured using the web-based Configuration Center:",[11,7064,7065],{},[92,7066],{"alt":7067,"src":7068},"Configuration Center Workflow setup","/images/blog/2022-03-01/editor_script_01.webp",[11,7070,7071,7072,7075],{},"Multiple Reactive Engines form a ",[246,7073,7074],{},"Reactive Cluster"," of their own. When setting up layline.io in a Cluster environment like Kubernetes, you actually set up a number of Nodes which then run Reactive Engines encapsulated in containers:",[11,7077,7078],{},[92,7079],{"alt":7080,"src":7081},"Reactive Engines within a Reactive Cluster within a Kubernetes Cluster","/images/blog/2022-03-01/19cc7f90.png",[11,7083,7084,7085,7088],{},"All Reactive Engines are created equal. They serve as execution contexts for Workflows. Simplified, you may view Workflows as equivalent to Microservices. The difference being that Workflows are ",[246,7086,7087],{},"configured"," and therefore a Configuration, and not programmed object code like with typical Microservices.",[11,7090,7091],{},[92,7092],{"alt":7093,"src":7094},"Zooming in to a Reactive Engine","/images/blog/2022-03-01/f9c28f84.png",[11,7096,7097],{},"Each Reactive Engine can run different Workflows (A, B and C above). Each Workflow can be dynamically instantiated multiple times. The number of instances is limited by how many resources a single Workflow instance consumes and how much resource is available and assigned to the execution context in which the Reactive Engine itself runs. In a Kubernetes Cluster this would be the image of a respective Pod:",[11,7099,7100],{},[92,7101],{"alt":7102,"src":7103},"Pod running a Container with a Reactive Engine","/images/blog/2022-03-01/ae038590.png",[11,7105,7106],{},"Any Reactive Engine can run any Workflow. Workflows are deployed either directly via the Configuration Center or via your preferred CI/CD tool (e.g. Bamboo et al).",[11,7108,7109],{},"This could result in a setup like this:",[11,7111,7112],{},[92,7113],{"alt":7114,"src":7115},"Workflow distribution within a Reactive Cluster","/images/blog/2022-03-01/4f40189e.png",[339,7117,7119],{"id":7118},"elastic-scaling","Elastic scaling",[11,7121,7122],{},"The number of instances of each Workflow can be scaled up and down dynamically, either by manual intervention from the Config Center or command line, or automatically based on data pressure.",[11,7124,7125],{},[92,7126],{"alt":7127,"src":7128},"Reactive Cluster scaling example","/images/blog/2022-03-01/368f3920.png",[11,7130,7131],{},"While the standard Kubernetes path to scale by firing up additional Pods remains valid, you can simply fire up additional Workflow instances within a Pod. Note that no additional Pods would be activated in this example, given that each Pod contains enough breathing room to scale. Because everything is scaled within a Reactive Engine, this process is extremely fast and efficient, requiring only few additional resources per instance and no intervention on Kubernetes level.",[339,7133,7135],{"id":7134},"advantage-resources","Advantage Resources",[11,7137,7138],{},"It makes sense to think, that one will require respective CPU power and RAM regardless of whether you distribute 30 Pods with the same Microservice on three Nodes, or three Reactive Engines with 10 instances of the same Workflow each on three Nodes. But that is not the case.",[11,7140,7141],{},"Depending on the characteristic of the Microservice which is replaced by a Workflow, you typically save between 25-50% of resources compared to the traditional way of deploying Microservices. It's also true, however, that a Reactive Engine running one Workflow instance only, requires a more resources than a custom Microservice which is only run one time.",[11,7143,7144],{},"It's a tradeoff between flexibility and resource requirements, which flips quickly to the favor of the layline.io model with the size of your processing scenario.",[11,7146,7147],{},[92,7148],{"alt":7149,"src":7150},"Resource requirements traditional Microservice vs. layline.io","/images/blog/2022-03-01/0c45546c.png",[339,7152,7154],{"id":7153},"advantage-setup","Advantage Setup",[11,7156,7157],{},"Setting up layline.io in a Kubernetes/Docker cluster means to deploy the same container on every Node. Each container runs a Reactive Engine and there are no other container types with different content. Just one.",[11,7159,7160],{},"For a Reactive Engine to know what Workflows to execute, a configuration is injected at runtime. Because all Reactive Engines form a Reactive Cluster of their own, it is enough to inject the Configuration into one Reactive Engine. It is then automatically distributed to all other Engines in the Cluster. Again, this can all be triggered manually or automatically through CI/CD tools.",[11,7162,7163],{},[92,7164],{"alt":7165,"src":7166},"Deploying workflow configurations into a Reactive Cluster","/images/blog/2022-03-01/e6bd8ff7.png",[11,7168,7169],{},"So unlike the pure Kubernetes/Docker concept, there is no hassle with different Pods containing different containers which need to be re-built on every change and then managed from a deployment point-of-view. Contrary to Kubernetes you don't have to think about where to run which Pod/Container upfront, or how to shuffle Pods around in case you want to rearrange them. In layline.io you can simply activate a preloaded Workflow in one or more Reactive Engines, or deploy a new Workflow Configuration to the Cluster in order to bring this Workflow online.",[26,7171,3743],{"id":3742},[1958,7173,7174,7185],{},[1961,7175,7176],{},[1964,7177,7178,7180,7183],{},[1967,7179,3893],{},[1967,7181,7182],{},"Kubernetes/Docker",[1967,7184,252],{},[1977,7186,7187,7198,7209,7220],{},[1964,7188,7189,7192,7195],{},[1982,7190,7191],{},"Scaling",[1982,7193,7194],{},"Scale via Pods",[1982,7196,7197],{},"Scale within Pod via Workflow instances",[1964,7199,7200,7203,7206],{},[1982,7201,7202],{},"Scaling reaction time",[1982,7204,7205],{},"medium",[1982,7207,7208],{},"fast",[1964,7210,7211,7214,7217],{},[1982,7212,7213],{},"Resource consumption",[1982,7215,7216],{},"Better with small scenarios",[1982,7218,7219],{},"Better with medium to large scenarios",[1964,7221,7222,7225,7228],{},[1982,7223,7224],{},"Setup",[1982,7226,7227],{},"Many different Pods and Containers",[1982,7229,7230],{},"Configurations injected into Reactive Engine",[11,7232,7233],{},"Kubernetes is great. Period. But there are downsides in complexity and operation.",[11,7235,7236],{},"layline.io provides a much better and leaner way to not only create and manage Services, but also to manage and distribute them in a Cluster environment. For medium to large scenarios, there is also a significant upside in regard to resource management and consumption.",[11,7238,7239,7240,7243,7244,7249],{},"You can download and use layline.io for free ",[226,7241,3092],{"href":3098,"rel":7242},[2187],". If you have any questions about layline.io please don't hesitate to ",[226,7245,7248],{"href":7246,"rel":7247},"https://layline.io/resources/contact",[2187],"contact us","!",[26,7251,4497],{"id":4496},[359,7253,7254,7260,7267,7272],{},[362,7255,7256],{},[226,7257,7259],{"href":3098,"rel":7258},[2187],"Download layline.io",[362,7261,7262],{},[226,7263,7266],{"href":7264,"rel":7265},"https://layline.io/blog/2022-02-14",[2187],"Fixing what's wrong with Microservices",[362,7268,3087,7269,1515],{},[226,7270,3092],{"href":3090,"rel":7271},[2187],[362,7273,3109,7274,1515],{},[226,7275,3113],{"href":3112},{"title":255,"searchDepth":256,"depth":256,"links":7277},[7278,7282,7283,7284,7290,7291],{"id":6977,"depth":256,"text":6978,"children":7279},[7280,7281],{"id":6981,"depth":540,"text":6982},{"id":7000,"depth":540,"text":7001},{"id":7013,"depth":256,"text":7014},{"id":7041,"depth":256,"text":7042},{"id":7051,"depth":256,"text":7052,"children":7285},[7286,7287,7288,7289],{"id":7055,"depth":540,"text":7056},{"id":7118,"depth":540,"text":7119},{"id":7134,"depth":540,"text":7135},{"id":7153,"depth":540,"text":7154},{"id":3742,"depth":256,"text":3743},{"id":4496,"depth":256,"text":4497},"2022-03-01","The traditional Microservices model on Kubernetes/Docker has some disadvantages which result in overly complex management and resource consumption. We explain the background and how layline.io can help.",{},"/blog/2022-03-01-kubernetes-workflows",{"title":6969,"description":7293},{"loc":7295},"blog/2022-03-01-kubernetes-workflows","F6EYoU3Mut1JYx6CZQtRPi44BqqFrF9q7ZSjRYagVDw",{"id":7301,"title":7302,"body":7303,"category":269,"date":7505,"description":7506,"extension":271,"featured":2934,"image":7405,"meta":7507,"navigation":272,"path":7508,"readTime":1340,"seo":7509,"sitemap":7510,"stem":7511,"__hash__":7512},"blog/blog/2022-02-22-event-driven.md","Event-driven or bust!? Are you missing out if your business is not event-driven?",{"type":8,"value":7304,"toc":7495},[7305,7309,7315,7321,7324,7332,7335,7338,7342,7345,7348,7352,7359,7365,7367,7375,7378,7382,7389,7400,7406,7409,7412,7415,7419,7422,7426,7429,7432,7435,7441,7445,7448,7451,7458,7465,7468,7479,7482,7484],[26,7306,7308],{"id":7307},"what-does-it-mean-to-be-data-driven-technically","What does it mean to be \"data-driven\" (technically)?",[11,7310,7311,7314],{},[246,7312,7313],{},"Data-driven"," means that data manifests somewhere in your business, and other systems - which may require this information for whatever purpose they fulfil - then poll the data from that source system. This could be by way of an ETL tool, a ready-made integration between receiving system and providing system, or simply some sort of custom-made polling interface.",[11,7316,7317],{},[92,7318],{"alt":7319,"src":7320},"Pull: Data-driven processing","/images/blog/2022-02-22/271495bb.png",[11,7322,7323],{},"Typical characteristics:",[604,7325,7326,7329],{},[362,7327,7328],{},"Synchronous data exchange (request / response pattern)",[362,7330,7331],{},"Trigger within minutes, hours, days, months",[11,7333,7334],{},"This is a very common architecture which can be found in basically all businesses. A typical example would be an ERP system which is then polled by an ETL system which in turn feeds a data warehouse. There are countless other \"data-driven\" examples of course.",[11,7336,7337],{},"Being data-driven is normal and for most companies the status-quo.",[26,7339,7341],{"id":7340},"are-you-missing-something-then-is-it-digital-transformation","Are you missing something then? Is it … Digital Transformation?",[11,7343,7344],{},"Are you a business that has spent a truckload of money on consultants telling you that you should undergo \"Digital Transformation\" or bust? You're not alone then. You don't need this info, however. Unless you're still working a mechanical cash register and keep a rolodex on your desk, you have already embarked on Digital Transformation a long time ago.",[11,7346,7347],{},"Digital Transformation is an evolutionary process. There is always an emergence of new technology and even newer technology which is just around the corner. Not all of this may be relevant to your business, but some technologies may be key to streamline your business further, create new offers and services, and improve on products and process. One of these trends in the recent past has been the notion of being \"event-driven\" and \"real-time\".",[26,7349,7351],{"id":7350},"event-driven-data-processing","Event-driven data processing",[11,7353,7354,7355,7358],{},"Being ",[246,7356,7357],{},"event-driven"," means to act on business events in real-time or near-real-time. One of the main characteristics, is that you get a constant stream of business \"events\" which are being pushed from the generating business infrastructure in real-time. Consumers of this data subscribe to the source and ingest the data. Data receipt is not acknowledged back to the source.",[11,7360,7361],{},[92,7362],{"alt":7363,"src":7364},"Push: Event-driven processing","/images/blog/2022-02-22/41136d3a.png",[11,7366,7323],{},[604,7368,7369,7372],{},[362,7370,7371],{},"Asynchronous data-exchange (react on events without reply)",[362,7373,7374],{},"Reaction within 0 to 3 seconds (roughly … should be milliseconds at best)",[11,7376,7377],{},"As you can see in this short comparison, there is a fundamental technical and architectural difference between data-driven and event-driven data consumption. It's fair to ask what is the point and where the value of this different type of architecture lies for your business. Let's look at this in a more general fashion:",[26,7379,7381],{"id":7380},"understanding-the-event-lifetime-value-elv","Understanding the Event-Lifetime-Value (ELV)",[11,7383,7384,7385,7388],{},"Based on the technical explanation above, you may wonder what this means for business and how it affects it. From a business-standpoint the real difference between event-driven and data-driven is the ",[246,7386,7387],{},"time-factor"," and how time can be monetized.",[11,7390,7391,7392,7395,7396,7399],{},"To distinguish the two you can argue that events which are being processed in a data-driven fashion (minutes/days/weeks/months) bear a ",[246,7393,7394],{},"strategic value"," which can actually grow over time (green wedge). Events which are processed in real-time on the other hand have an ",[246,7397,7398],{},"operational value"," which decreases rapidly as the data ages within seconds (orange wedge).",[11,7401,7402],{},[92,7403],{"alt":7404,"src":7405},"Capturing the Event-Lifetime-Value (ELV)","/images/blog/2022-02-22/1d4aa019.png",[11,7407,7408],{},"In other words: The faster (real-time) your reaction to the information, the higher the potential operational value. In contrary, the more data you gather, the higher the strategic value of the aggregate may be. Stock prices are a perfect example here in that lightning fast information about price movements bears a very high potential to make the better trade before others do (speed of information). In the longer run, time-series data about specific stock price movement helps to gain better insights about trends and correlations of a particular stock.",[11,7410,7411],{},"Of course, this is a simplified view as both event-driven and data-driven timing may overlap. It all depends on your business use-cases as well as how you can take advantage of real-time information.",[11,7413,7414],{},"A key point here is that up to now many businesses do not use real-time data at all. For them this may be an entirely new value which can be transformed into a valued-add.",[26,7416,7418],{"id":7417},"alas-should-you-care-about-event-driven-data-processing","Alas, should you care about event-driven data processing?",[11,7420,7421],{},"There is no simple answer to this. It depends on your business model, and whether there is any identifiable benefit had you a lot more data available in real-time. It would be very neglectful to not fully understand what you may be missing out on, however. It's here where it makes sense to take a good look around and see where markets and technology are headed, how others are benefiting from it and what it means to you.",[26,7423,7425],{"id":7424},"relevant-drivers-spawning-mega-market-growth","Relevant drivers spawning mega market growth",[11,7427,7428],{},"Digital Transformation is truly transforming the world. This is about the integration of intelligent data into everything that we do. It is about an event- and data-driven world which is always-on, always tracking, always learning and reacting to these findings.",[11,7430,7431],{},"Of course, it is data which is the lifeblood of Digital Transformation, and without it not one of the changes in this arena we encounter today and tomorrow would happen. As people, industries and things get more and more connected, more data is available. In turn more value can be harvested from it and new services can be created. This momentum propels data generation and consumption into ever more heights, and it is predicted, that global data generation will grow 28% YoY over the next five years (source: datanami). This growth and the requirements of Digital Transformation put an astronomical strain on organizations. If they haven't already done so, they need to not only need rethink their business strategy, but also how to deliver on it technically.",[11,7433,7434],{},"There is no doubt we are looking at literal explosion of data volume spawned by technical advancements, which in turn feed new fires over the next three to six years. 5G has started to roll out, allowing for a huge number of new use cases due to higher bandwidth and extremely low latency. It is estimated that by 2025 we will have 150 billion connected devices and that up to 5,000 digital interactions per day will be attributable to each person. With this grows the datasphere - and especially real-time data volume - is posed to grow up to ten times within the next few years.",[11,7436,7437],{},[92,7438],{"alt":7439,"src":7440},"Mega data-growth and business opportunity","/images/blog/2022-02-22/373e1554.png",[26,7442,7444],{"id":7443},"what-is-your-answer","What is your answer?",[11,7446,7447],{},"We understand that legacy systems are not the answer. Some simple, but serious problems are that they are just not designed for these data masses, are usually not cloud enabled, do not scale, and are the opposite of agile. Getting to grips with this is life-threatening to many, and a real competitive problem for most.",[11,7449,7450],{},"One of the main challenges in this setting is how to integrate with all the data sources and sinks quickly, make sense of the information, and ensure that data is handled with low latency and at truly massive scale.",[11,7452,7453,7454,7457],{},"For this purpose layline.io has taken a new approach and put together a brand-new way of solving this by introducing its ",[246,7455,7456],{},"Reactive Data Integration"," solution.",[11,7459,7460,7461,7464],{},"It connects to virtually anything and can interpret, process, forward and interact with everything on all levels and at the speed of data. Individually configurable workflows fulfil individual tasks. By way of its architecture, tens, hundreds, even thousands of layline.io Reactive Engines can automatically be deployed natively or as microservices (via docker & kubernetes) across an array of nodes. Forming an actual ",[246,7462,7463],{},"Reactive Data Integration Mesh",". These nodes can range from very small devices on the edge up to large scale installations in the core data center (on-premise or cloud). Because the engines on the nodes are aware of each other, they can also look out for each other. Load deviations and failures are automatically balanced and neutralized.",[11,7466,7467],{},"In short: It's a non-stop, immortal, transparent data network which scales in three dimensions:",[604,7469,7470,7473,7476],{},[362,7471,7472],{},"From core to edge",[362,7474,7475],{},"Across nodes",[362,7477,7478],{},"Even inside a node,",[11,7480,7481],{},"and allows business to quickly configure the logic they need when and where they need it, abstracting the underlying infrastructure almost completely.",[26,7483,4497],{"id":4496},[359,7485,7486,7491],{},[362,7487,3087,7488,1515],{},[226,7489,3092],{"href":3090,"rel":7490},[2187],[362,7492,3109,7493,1515],{},[226,7494,3113],{"href":3112},{"title":255,"searchDepth":256,"depth":256,"links":7496},[7497,7498,7499,7500,7501,7502,7503,7504],{"id":7307,"depth":256,"text":7308},{"id":7340,"depth":256,"text":7341},{"id":7350,"depth":256,"text":7351},{"id":7380,"depth":256,"text":7381},{"id":7417,"depth":256,"text":7418},{"id":7424,"depth":256,"text":7425},{"id":7443,"depth":256,"text":7444},{"id":4496,"depth":256,"text":4497},"2022-02-22","You may be missing out if you are data-driven only. In fact, every business is data-driven. But you should ask yourself what that really means, and whether it is sufficient for you tomorrow.",{},"/blog/2022-02-22-event-driven",{"title":7302,"description":7506},{"loc":7508},"blog/2022-02-22-event-driven","-n2rDHrNjOneTebE_glaPnMU5p56Jqx5n6fOcc0m73I",{"id":7514,"title":7266,"body":7515,"category":269,"date":7752,"description":7753,"extension":271,"featured":2934,"image":7629,"meta":7754,"navigation":272,"path":7755,"readTime":1605,"seo":7756,"sitemap":7757,"stem":7758,"__hash__":7759},"blog/blog/2022-02-14-microservices.md",{"type":8,"value":7516,"toc":7744},[7517,7520,7528,7532,7536,7554,7558,7601,7604,7608,7611,7624,7630,7635,7641,7664,7667,7673,7691,7694,7700,7703,7706,7709,7712,7718,7720],[11,7518,7519],{},"For some years now Microservices and Service-oriented architectures have been all the rage. Shortly after that containerization helped to abstract installed platform from deployed platform by packaging OS and dependent libs up with the actual application.",[11,7521,7522,7523,7527],{},"But where there is light, there is shadow. Working with and using Microservices comes with its own set of challenges. We found a pretty comprehensive list ",[226,7524,3092],{"href":7525,"rel":7526},"https://www.toolbox.com/tech/data-management/articles/top-10-challenges-of-using-microservices-for-managing-distributed-systems/",[2187],". Let's look at some of main pros and cons:",[26,7529,7531],{"id":7530},"main-challenges-of-microservice-development-deployment-and-operation","Main challenges of Microservice development, deployment and operation",[339,7533,7535],{"id":7534},"the-good","The Good",[604,7537,7538,7544,7549],{},[362,7539,7540,7543],{},[246,7541,7542],{},"Atomicity",": Autonomous services can be treated individually in terms of development and execution (apart from interfaces).",[362,7545,7546,7548],{},[246,7547,2977],{},": Individual services are usually not compromised when it comes to failures in other services which leads to better resilience.",[362,7550,7551,7553],{},[246,7552,2971],{},": Individual services can be scaled elastically on-demand.",[339,7555,7557],{"id":7556},"the-bad","The Bad",[604,7559,7560,7566,7572,7578,7584,7589,7595],{},[362,7561,7562,7565],{},[246,7563,7564],{},"Loose coupling",": Microservices usually do not know about each other and their broader execution context. They are atomic in nature. Communication between them comes with an overhead and is not standardized.",[362,7567,7568,7571],{},[246,7569,7570],{},"Monitoring",": Comprehensive monitoring of a variety of Microservices is extremely hard and almost impossible to do. Clearly uncovering individual problems across different services can be extremely hard due to different types of logs spread all over the place, unclear interdependencies between services and transaction spanning of services etc.",[362,7573,7574,7577],{},[246,7575,7576],{},"Debugging",": Errors which are occurring in a complex distributed Microservices architecture can be extremely time-consuming and expensive to trace. There is no overarching monitoring system, but rather individual logs and stack traces which need to be investigated in order to safely conclude what the cause of the error was.",[362,7579,7580,7583],{},[246,7581,7582],{},"Security",": An integral characteristic of Microservices are their interfaces/APIs. Especially in distributed environments each of them requires special care in regard to security. It's easy to lose oversight and control in such complex framework environments.",[362,7585,7586,7588],{},[246,7587,2977],{},": With lots of different types of Microservices, which may be developed by different teams, it becomes exponentially harder to ensure proper failover mechanisms, so that the whole system can act accordingly when one or more the Microservices fail.",[362,7590,7591,7594],{},[246,7592,7593],{},"Deployment",": Deployment of individual Microservices in a complex setup without downtime is hard to orchestrate and at times impossible to accomplish without restarting everything.",[362,7596,7597,7600],{},[246,7598,7599],{},"Communication",": There has to be some form of standardization of communication between Microservices in terms of serialization, security, request options, error handling and the list of expected responses. Some form of top-level design orchestration is very necessary or else will result in failed communication and latency issues.",[11,7602,7603],{},"This is just to name a few of the challenges. There are many more challenges when it comes maintenance, networking, team management etc. as you can imagine.",[26,7605,7607],{"id":7606},"solution-microconfigurations-instead-of-microservices","Solution: MicroConfigurations instead of MicroServices",[11,7609,7610],{},"While the idea of Microservices is great, it can get very nasty very fast. Naturally the question is whether there is a way to keep the good parts of a Microservices setup and avoid the bad parts.",[11,7612,7613,7616,7617,7619,7620,7623],{},[246,7614,7615],{},"Micro-Configurations"," may be able to help here. We define Micro-Configuration (or MicroConfig) as a separation between actual service logic (the ",[246,7618,6473],{},") and service execution (the ",[246,7621,7622],{},"Engine",").",[11,7625,7626],{},[92,7627],{"alt":7628,"src":7629},"MicroService vs. MicroConfig","/images/blog/2022-02-14/482fddcf.png",[11,7631,7632,7634],{},[246,7633,6473],{}," in this context shall be purely limited to the business logic, i.e. what to do with the data, what action to trigger, etc.",[11,7636,7637,7640],{},[246,7638,7639],{},"Execution"," is everything that makes execution of the Configuration possible:",[359,7642,7643,7646,7649,7652,7655,7658,7661],{},[362,7644,7645],{},"execution of the logic (configuration),",[362,7647,7648],{},"centralized and standardized logging of activity and problems,",[362,7650,7651],{},"orchestration of execution across services",[362,7653,7654],{},"metrics reporting for monitoring",[362,7656,7657],{},"debugging support",[362,7659,7660],{},"standardised communication across service boundaries",[362,7662,7663],{},"and much more.",[11,7665,7666],{},"This generally isn't a new concept in the world of tech.",[11,7668,7669,7672],{},[246,7670,7671],{},"Example",": A typical database distinguishes between database structure (tables, indexes, constraints, etc.) and database engine (interpretation, storing, serving). While the engine is the same for every user, the structure is unique. Yet nobody would entertain the idea to hard code tables right into the engine. Separation of config and engine is what makes the database generic in the first place, each of them having a special purpose and power.",[11,7674,7675,7676,7679,7680,7683,7684,7687,7688,7690],{},"layline.io is similar in that Services are implemented as so-called ",[246,7677,7678],{},"Workflow Configurations"," not as monolithic executables. An unlimited number of different Workflows can be defined. Workflows are run by ",[246,7681,7682],{},"Reactive Engines"," which in turn run on ",[246,7685,7686],{},"Nodes",". A Kubernetes Pod or a Raspberry Pi for example would be a Node. Two or more Engines form a logical ",[246,7689,7074],{},". The setup solely depends on your requirements and environment. A theoretically indefinite number of Engines (on Nodes) can be spawned and run in a geographically distributed logical cluster. Edge-computing is one of the interesting use-cases here. Because everything runs on the same type of Reactive Engine you don't have to think about what to deploy on a physical level.",[11,7692,7693],{},"Deployment of Workflow Configurations happens automatically in that a configuration is published to a Node in the Reactive Cluster, and the Cluster then automatically propagates the configuration throughout the Cluster. This avoids having to worry about low-level physical deployment of Microservices to Pods or actual physical Nodes. New nodes being added to the Cluster also automatically receive the configuration data and can commence processing immediately.",[11,7695,7696],{},[92,7697],{"alt":7698,"src":7699},"Deployment of Configurations and Auto-Distribution","/images/blog/2022-02-14/ec411c06.png",[11,7701,7702],{},"Resilience, Scalability and Failover is built-in to layline.io. Constant Cluster monitoring ensures compliance with configured scales of Workflows instances, and automatically rebalances the workload of Workflows in case of Node-failure.",[11,7704,7705],{},"Central monitoring and logging ensures that problems within the Reactive Cluster are immediately spotted. Remedy can be commonly provided through the UI without interference on physical level.",[11,7707,7708],{},"Everything within the layline.io platform is standardized on execution level, but open on configuration level. This allows you to set up what you need, without having to worry about the hard parts on how the infrastructure actually runs.",[11,7710,7711],{},"While the concept of a framework may not be what the hard-core \"low-level-only\" developer likes to embrace, it does make a lot of sense technically and business-wise. You wouldn't program your own database either, or would you?",[11,7713,7714],{},[92,7715],{"alt":7716,"src":7717},"Microservices vs. layline.io","/images/blog/2022-02-14/994ca5a9.png",[26,7719,4497],{"id":4496},[359,7721,7722,7728,7735,7740],{},[362,7723,7724],{},[226,7725,7727],{"href":7525,"rel":7726},[2187],"Top 10 Challenges of Using Microservices for Managing Distributed Systems",[362,7729,7730],{},[226,7731,7734],{"href":7732,"rel":7733},"https://www.toolbox.com/tech/devops/guest-article/heres-why-microservices-desperately-need-service-mesh-anomaly-detection/",[2187],"Here's Why Microservices Desperately Need Service Mesh Anomaly Detection",[362,7736,3087,7737,1515],{},[226,7738,3092],{"href":3090,"rel":7739},[2187],[362,7741,3109,7742,1515],{},[226,7743,3113],{"href":3112},{"title":255,"searchDepth":256,"depth":256,"links":7745},[7746,7750,7751],{"id":7530,"depth":256,"text":7531,"children":7747},[7748,7749],{"id":7534,"depth":540,"text":7535},{"id":7556,"depth":540,"text":7557},{"id":7606,"depth":256,"text":7607},{"id":4496,"depth":256,"text":4497},"2022-02-14","For some years now Microservices and Service-oriented architectures have been all the rage. But there are downsides. Can they be overcome?",{},"/blog/2022-02-14-microservices",{"title":7266,"description":7753},{"loc":7755},"blog/2022-02-14-microservices","XQujU147ux_0fSlx-s3XzjcHOk_9eNTkCcjksPT-mvA",{"id":7761,"title":7762,"body":7763,"category":269,"date":7892,"description":7893,"extension":271,"featured":2934,"image":7894,"meta":7895,"navigation":272,"path":7896,"readTime":3643,"seo":7897,"sitemap":7898,"stem":7899,"__hash__":7900},"blog/blog/2022-02-08-data-pressure.md","Dealing with data-pressure in message-based systems",{"type":8,"value":7764,"toc":7886},[7765,7769,7772,7775,7785,7788,7792,7795,7798,7801,7809,7812,7815,7834,7840,7849,7855,7859,7862,7868,7870],[26,7766,7768],{"id":7767},"what-is-data-pressure","What is data-pressure?",[11,7770,7771],{},"You hear a lot about data-pressure when it comes to non-stop systems. What is it and why is it so important?",[11,7773,7774],{},"Pressure in the physical sense describes an imbalance between gas or fluid between two confined compartments. It goes both ways until an equilibrium is reached. If you want to manage it, you usually put a valve between the two.",[11,7776,7777,7778,2337,7781,7784],{},"In data processing systems ",[246,7779,7780],{},"data-pressure",[246,7782,7783],{},"upstream-pressure"," describes the amount of data which is ready for processing. For file based (batch) solutions this simply describes the amount of files which are waiting to be processed (asynchronous). Processing speed is purely based on the processing power of the downstream actors and is demand based. Data-pressure in batch usually is no threat to system overload because it is implicit. The batch processing system will always only process as much as it can.",[11,7786,7787],{},"It's a very different story in modern, message-driven, real-time processing environments, however, where data-pressure is explicit because data needs to be processed as it arrives.",[26,7789,7791],{"id":7790},"importance-of-back-pressure-in-non-stop-real-time-systems","Importance of back-pressure in non-stop real-time systems",[11,7793,7794],{},"Message-driven use-cases usually require that data should be handled in real-time, at all times. Therefore, systems have to be able to scale elastically in order to handle peak loads, or free up unneeded resources during low data-pressure windows.",[11,7796,7797],{},"There are countless examples of architectures which clog-up when dealing with large data loads. This often results in a vicious-cycle which typically leads to cardiac-arrest of such an architecture. The main conundrum is a missing negative demand-signal (or high data back-pressure-signal) to upstream actors upon which they can react. If there were such a signal, appropriate actions could be taken.",[11,7799,7800],{},"Such actions could be:",[359,7802,7803,7806],{},[362,7804,7805],{},"slowing processing down overall up the data stream, or",[362,7807,7808],{},"spawning more processing power to handle the added pressure",[11,7810,7811],{},"Once upstream data-pressure decreases, the counter-measures can be reversed. More data can again be delivered, or previously activated processing power can be decommissioned.",[11,7813,7814],{},"So to summarize, we have:",[604,7816,7817,7826],{},[362,7818,7819,7820,6354,7823,7825],{},"a ",[246,7821,7822],{},"data-signal",[246,7824,7783],{}," which signals that data is available for processing, and we have",[362,7827,7819,7828,6354,7831,7833],{},[246,7829,7830],{},"demand-signal",[246,7832,2864],{}," which signals how loaded the downstream actors are, and whether pressure from upstream actors can be relieved on to downstream actors.",[11,7835,7836],{},[92,7837],{"alt":7838,"src":7839},"Data-pressure Reactivity","/images/blog/2022-02-08/e436ddd5.png",[11,7841,7842,7843,7848],{},"Using these signals, the system is able to negotiate an equilibrium between all participants which ensures that processing never stops, but rather slows down (or additional capacity is automatically made available). This problem is well recognized and defined in the ",[226,7844,7847],{"href":7845,"rel":7846},"https://www.reactivemanifesto.org/",[2187],"Reactive Manifesto"," which requires systems to be message-driven, elastic and resilient, and therefore responsive to load. Systems which cater to these requirements are called \"Reactive\".",[11,7850,7851],{},[92,7852],{"alt":7853,"src":7854},"Reactive Manifesto: Means - Form - Value","/images/blog/2022-02-08/f8cdb1ba.png",[26,7856,7858],{"id":7857},"how-laylineio-handles-it","How layline.io handles it",[11,7860,7861],{},"It sounds like the solution to the back-pressure challenge is simple. But it's actually hard to solve since all participants in this dance need to be data-pressure aware, both ways. Reactive stream management has solved this problem which is why layline.io takes full advantage it under the hood. It's not for the faint-of-heart, however, and comes with a steep learning and experience curve attached. layline.io shields its users from this complexity in an easy-to-use platform, which provides all production necessary features like UI-driven low-code configurability, one-click deployment, monitoring and much more.",[11,7863,7864],{},[92,7865],{"alt":7866,"src":7867},"layline.io Project Configuration","/images/blog/2022-02-08/project_workflow_04.webp",[26,7869,4497],{"id":4496},[359,7871,7872,7877,7882],{},[362,7873,7874],{},[226,7875,7847],{"href":7845,"rel":7876},[2187],[362,7878,3087,7879,1515],{},[226,7880,3092],{"href":3090,"rel":7881},[2187],[362,7883,3109,7884,1515],{},[226,7885,3113],{"href":3112},{"title":255,"searchDepth":256,"depth":256,"links":7887},[7888,7889,7890,7891],{"id":7767,"depth":256,"text":7768},{"id":7790,"depth":256,"text":7791},{"id":7857,"depth":256,"text":7858},{"id":4496,"depth":256,"text":4497},"2022-02-08","How to deal with data pressure in non-stop message-driven solutions and ensure non-stop uptime under load.","/images/blog/2022-02-08/lucas-van-oort-_FjIWDrtfmU-unsplash.webp",{},"/blog/2022-02-08-data-pressure",{"title":7762,"description":7893},{"loc":7896},"blog/2022-02-08-data-pressure","FPh-_hJrg0GeJYS1iBP910cvhxYTu4RaFdrmlWGBa68",1776673530138]