<p>RabbitMQ is a robust, open-source message broker that facilitates communication between applications by sending and receiving messages. Whether you're building a microservices architecture or a distributed system, RabbitMQ ensures reliable, scalable, and asynchronous messaging. In this blog, we’ll walk through its core components and concepts, from producers to consumers, and dive into some advanced features like round-robin dispatching and virtual hosts.</p><p><img src="https://storage.googleapis.com/personal-blog-darmajr.appspot.com/blog-content/1742090760730_rabbit-mg-steps.png" alt="rabbit mq floq" width="720px"></p><h1><strong>1. The Producer: Where It All Begins</strong></h1><p>In RabbitMQ, the <strong>producer</strong> is the entity (e.g., an application or service) that generates and sends messages. It doesn’t directly deliver messages to queues—instead, it publishes them to an exchange. Think of the producer as a post office clerk dropping letters into a sorting system rather than delivering them to mailboxes.</p><p>Producers connect to RabbitMQ via a client library (available in languages like Python, Java, or Node.js) and specify the message content, routing details, and exchange to use.</p><p>Once a producer has a message—say, a JSON object like {"order_id": 123, "status": "pending"}—it <strong>publishes</strong> it to RabbitMQ. The message isn’t just free-floating; it’s sent to an <strong>exchange</strong>, a key component that decides where the message goes next. Publishing is typically asynchronous, meaning the producer doesn’t wait for confirmation unless explicitly configured (e.g., with publisher confirms for reliability).</p><p>Messages can include metadata like headers or priority levels, but the core payload is what drives the system.</p><h1><strong>2. Exchange: The Message Router</strong></h1><p>The <strong>exchange</strong> is RabbitMQ’s routing engine. It receives messages from producers and forwards them to queues based on rules called <strong>bindings</strong>. RabbitMQ supports several exchange types, each with unique routing logic:</p><ul><li><strong>Direct Exchange</strong>: Routes messages to queues based on an exact match between the message’s routing key (e.g., "order.created") and the queue’s binding key. Ideal for unicast scenarios.</li><li><strong>Fanout Exchange</strong>: Ignores routing keys and broadcasts messages to all bound queues. Perfect for pub/sub patterns where every subscriber gets the message.</li><li><strong>Topic Exchange</strong>: Uses pattern matching on routing keys (e.g., "order.<em>" or "</em>.created") to route messages to queues. Flexible for hierarchical or wildcard-based routing.</li><li><strong>Header Exchange</strong>: Routes based on message header attributes rather than routing keys. Less common but useful for complex metadata-driven routing.</li></ul><p>Each exchange type serves a specific purpose, making RabbitMQ adaptable to diverse use cases.</p><h1><strong>3. Binding: Connecting Exchanges to Queues</strong></h1><p>A <strong>binding</strong> is the link between an exchange and a queue. It defines the rules for how messages flow. For example:</p><ul><li>In a direct exchange, a binding might say, “Send messages with routing key ‘error’ to Queue A.”</li><li>In a topic exchange, a binding could be “Send messages matching ‘*.log’ to Queue B.”</li></ul><p>Bindings are configured by the application or administrator, ensuring messages reach the right destination based on the exchange’s logic.</p><h1><strong>4. Queues: The Message Holders</strong></h1><p><strong>Queues</strong> are where messages land after being routed by the exchange. They act as buffers, storing messages until a consumer retrieves them. Queues are durable (survive broker restarts) or transient, and they can have properties like message TTL (time-to-live) or maximum length.</p><p>A queue can be bound to multiple exchanges, and multiple queues can receive messages from the same exchange, depending on the binding rules.</p><h1><strong>5. Consume Message: The Consumer’s Role</strong></h1><p>The <strong>consumer</strong> is the application or service that retrieves messages from a queue and processes them. Consumers can operate in two modes:</p><ul><li><strong>Push</strong>: RabbitMQ delivers messages to the consumer as they arrive (using a subscription model).</li><li><strong>Pull</strong>: The consumer explicitly requests messages from the queue.</li></ul><p>Once a message is consumed, the consumer acknowledges it (manual or automatic ACK), telling RabbitMQ it’s been processed. If unacknowledged, the message can be requeued for another consumer—ensuring no data is lost.</p><h2><strong>Extra Topic 1: Diagram of Round Robin Dispatching</strong></h2><p>RabbitMQ uses <strong>round-robin dispatching</strong> to distribute messages fairly among multiple consumers subscribed to the same queue. Here’s how it works:</p><p>Imagine a queue with three consumers (C1, C2, C3) and five messages (M1, M2, M3, M4, M5). RabbitMQ delivers them like this:</p><ul><li>M1 → C1</li><li>M2 → C2</li><li>M3 → C3</li><li>M4 → C1</li><li>M5 → C2</li></ul><p>This ensures load balancing across consumers, assuming they’re all available and processing at similar rates. You can tweak this with prefetch settings (e.g., basic.qos) to limit how many unacknowledged messages a consumer handles at once.</p><h2><strong>Extra Topic 2: Virtual Hosts</strong></h2><p>A <strong>virtual host</strong> (vhost) in RabbitMQ is a logical separation within a single broker instance. Think of it as a tenant in a multi-tenant system. Each vhost has its own set of exchanges, queues, bindings, and permissions, isolated from others.</p><p>For example:</p><p>- Vhost /app1 might handle order processing.</p><p>- Vhost /app2 might manage user notifications.</p><p>Admins create vhosts via the RabbitMQ management interface or API, assigning users specific access rights. This isolation enhances security and organization, especially in shared environments.</p><h1><strong>Conclusion</strong></h1><p>RabbitMQ’s architecture—producers publishing to exchanges, exchanges routing via bindings to queues, and consumers processing messages—makes it a versatile tool for messaging needs. Features like round-robin dispatching ensure fair workload distribution, while virtual hosts provide logical separation for complex systems. Whether you’re broadcasting updates with fanout or filtering logs with topic exchanges, RabbitMQ has you covered.</p>

RabbitMQ is a robust, open-source message broker that facilitates communication between applications by sending and receiving messages. Whether you're building a microservices architecture or a distributed system, RabbitMQ ensures reliable, scalable, and asynchronous messaging. In this blog, we’ll walk through its core components and concepts, from producers to consumers, and dive into some advanced features like round-robin dispatching and virtual hosts.

Understanding RabbitMQ: A Favorite Simple Messaging Service!

<h2><strong>Introduction: A New Way to Work with AI</strong></h2><p>What if your AI assistant could do exactly what you need it to—like count words in a sentence, fetch live data, or even control smart devices—without waiting for the developers to add those features? That’s where the <strong>Model Context Protocol (MCP)</strong> comes in. It’s a powerful new framework from Anthropic that lets you extend AI models like Claude with custom tools you build yourself. In this blog, I’ll break down the concept of MCP, share how I used it to create a simple word-counting tool for Claude Desktop, and explore why this technology is so exciting for the future of AI.</p><h3><strong>What is MCP? A Bridge Between AI and Your Ideas</strong></h3><p>The Model Context Protocol, or MCP, is like a universal translator that lets AI models (like Claude) talk to external tools you create. Imagine Claude as a super-smart assistant who can follow instructions—but only knows what’s built into it. MCP acts as a bridge, allowing Claude to send requests to your custom tool (say, a word counter) and get answers back, all in a standardized way.</p><p>Here’s the basic idea:</p><ul><li><strong>Claude (the AI)</strong> sends a message saying, “Hey, can you do this task for me?”</li><li><strong>Your Tool (the server)</strong> listens for that message, does the work, and sends the result back.</li><li><strong>The Communication</strong> happens through a simple format called JSON-RPC, using basic input/output channels (like a terminal’s input and output).</li></ul><p>For example, I wanted Claude to count words in a sentence. With MCP, I built a small server that listens for Claude’s request, counts the words, and tells Claude the answer—like, “The text has 5 words.” It’s that straightforward!</p><h2><strong>Why MCP is a Big Deal</strong></h2><p>So, why should you care about MCP? Here are a few reasons it’s a game-changer:</p><ul><li><strong>Make AI Your Own</strong>: You’re no longer stuck with what Claude can do out of the box. Want it to analyze your local files? Check stock prices? Turn on your smart lights? With MCP, you can make it happen by writing your own tools.</li><li><strong>Works with Any Language</strong>: Whether you prefer Python, JavaScript, or another language, MCP doesn’t care—it’s designed to be flexible.</li><li><strong>Endless Possibilities</strong>: From simple tasks like my word counter to complex integrations (think connecting to APIs or databases), MCP lets you dream big.</li></ul><p>When I got MCP working with Claude Desktop, I was amazed at how easy it was to teach Claude something new. I just told it, “Count the words in: hello world,” and it responded, “The text has 2 words.” That moment felt like magic—it showed me how MCP can turn AI into a truly personal assistant.</p><h2><strong>How Does MCP Work? The Big Picture</strong></h2><p>At a high level, MCP connects Claude to your tool in three main steps:</p><ol><li><strong>The Handshake</strong>: When your tool starts, Claude sends a message called initialize to say hello and ask what your tool can do. Your tool responds by listing its capabilities—like, “I can count words.”</li><li><strong>The Request</strong>: When you ask Claude to do something (e.g., “Count the words in this sentence”), it sends a message to your tool with the task details.</li><li><strong>The Response</strong>: Your tool processes the request, does the work (like counting words), and sends the answer back to Claude, which then shares it with you.</li></ol><p>This all happens behind the scenes, so to you, it just looks like Claude suddenly gained a new skill. In my case, I set up a Python-based server for Claude Desktop, and once it was running, Claude could count words as if it were a built-in feature.</p><h2><strong>My Journey: Building a Word Counter with MCP</strong></h2><p>To see MCP in action, I decided to build a simple word-counting tool for Claude Desktop. The idea was straightforward: I’d ask Claude to count the words in a sentence, and my tool would do the work. Here’s what I learned from the experience:</p><ul><li><strong>It’s Empowering</strong>: Teaching Claude to count words felt like giving it a superpower. Suddenly, I could imagine all sorts of other tools I could build.</li><li><strong>It’s Accessible</strong>: You don’t need to be a coding expert to get started. I used Python because it’s beginner-friendly, but the concept is the same no matter what language you choose.</li><li><strong>It’s Fun</strong>: There’s something thrilling about seeing Claude use your tool for the first time. When I typed “Count the words in: hello world” and got “The text has 2 words” back, I couldn’t stop smiling.</li></ul><h2><strong>What’s Next for MCP? Imagine the Possibilities</strong></h2><p>My word counter is just a starting point. MCP opens the door to so many exciting ideas:</p><ul><li><strong>Productivity Boosters</strong>: Build a tool to summarize your local documents or fetch your calendar events.</li><li><strong>Creative Helpers</strong>: Create a tool that generates random writing prompts or analyzes the tone of your text.</li><li><strong>Smart Integrations</strong>: Connect Claude to live data—like weather updates or stock prices—or even control smart home devices.</li></ul><p>The best part? MCP isn’t just for one person. Developers can share their tools with the community, creating a library of features anyone can add to Claude. Imagine downloading a “weather reporter” tool or a “code debugger” tool with a single click—that’s the future MCP could enable.</p><h2><strong>Conclusion: Join the MCP Revolution</strong></h2><p>The Model Context Protocol is more than just a tech concept—it’s a way to make AI truly yours. By building a simple word-counting tool for Claude Desktop, I got a glimpse of how MCP can transform the way we interact with AI. It’s easy to get started, incredibly powerful, and opens up a world of creativity.</p><p>If you’re curious to try MCP yourself, Anthropic’s <a href="https://modelcontextprotocol.io/quickstart/server" rel="noopener noreferrer" target="_blank" style="color: rgb(0, 102, 204);"><strong>MCP Quickstart Guide</strong></a> is a great place to begin. Whether you’re a developer or just someone who loves tinkering with tech, MCP lets you take AI to the next level. What will you build for Claude? I’d love to hear your ideas—drop them in the comments below!</p>

What if your AI assistant could do exactly what you need it to—like count words in a sentence, fetch live data, or even control smart devices—without waiting for the developers to add those features? That’s where the Model Context Protocol (MCP) comes in. It’s a powerful new framework from Anthropic that lets you extend AI models like Claude with custom tools you build yourself. In this blog, I’ll break down the concept of MCP, share how I used it to create a simple word-counting tool for Claude Desktop, and explore why this technology is so exciting for the future of AI.

Supercharge Your AI with MCP: The Future of Custom AI Tools

<h2>What Is a Circuit Breaker?</h2><p>Imagine you’re using electricity at home, and a short circuit occurs. The <em>circuit breaker</em> in your electrical panel cuts the power to prevent a fire. In software, the concept is similar: it’s a design pattern that protects your system from repeated failures when calling external services (APIs, databases, etc.).</p><h2>Main Purposes:</h2><h3><strong>Detect Failures</strong></h3><p>The first job of a <em>circuit breaker</em> is to act as a vigilant watchdog, constantly monitoring interactions between your application and external services like APIs, databases, or third-party systems. It keeps an eye on every request, tracking whether they succeed or fail based on specific criteria, such as receiving an error code (e.g., HTTP 500), timing out after a set duration (e.g., no response within 2 seconds), or encountering exceptions like network disconnections.</p><p>To do this effectively, the circuit breaker collects data over a defined window—perhaps the last 10 requests or the past 30 seconds—and calculates metrics like the total number of failures or the failure rate (e.g., 60% of calls failed). If these metrics cross a configurable threshold—say, five failures in a row or a 50% error rate—it recognizes that something’s wrong with the external service. This detection isn’t just about noticing a single hiccup; it’s about identifying patterns of unreliability that could harm your system if left unchecked. By catching these issues early, the circuit breaker ensures your application doesn’t blindly keep trying a service that’s clearly struggling.</p><h3><strong>Prevent Cascading Failures</strong></h3><p>Once a failure is detected, the circuit breaker steps in to stop a domino effect known as <em>cascading failures</em>, where one broken component drags down the entire system. Imagine an e-commerce app where the payment API is down: without a circuit breaker, every user request might hang, waiting for a timeout, piling up server resources, slowing the database, and eventually crashing the whole application.</p><p>In its Closed state, the circuit breaker allows calls to proceed, but as soon as failures hit the threshold, it flips to Open, cutting off all further attempts to contact the faulty service. This immediate halt prevents the problem from rippling through your system—your app stops wasting threads, memory, or CPU cycles on a hopeless task. Instead of letting a single point of failure—like a slow third-party API—overload your servers or exhaust connection pools, the circuit breaker isolates the issue, keeping the rest of your application stable and responsive. It’s like closing a floodgate to protect the town downstream from a burst dam.</p><h3><strong>Provide a Fallback Response</strong></h3><p>When the circuit breaker blocks calls in its Open state, it doesn’t just leave users hanging—it offers a fallback response to keep the system usable. This fallback is a preplanned alternative to the failed service’s output, designed to minimize disruption.</p><p>For example, if a weather API fails, the circuit breaker might return a cached forecast from an hour ago or a simple message like "Weather data unavailable, try again later." In a payment system, it could redirect users to an alternative checkout method or log the attempt for later retry. The fallback doesn’t fix the root problem, but it ensures graceful degradation.</p><p>Your application keeps running in a limited capacity rather than crashing or showing cryptic errors. Crafting a good fallback requires understanding your use case: it might be static data, a default value, or even a call to a backup service. By providing this safety net, the circuit breaker maintains user trust and buys time for the external service to recover without sacrificing functionality entirely.</p><p><img src="https://storage.googleapis.com/personal-blog-darmajr.appspot.com/blog-content/1741949039277_Circuit-Breaker-Pattern.jpg" alt="Circuit Breaker Pattern" width="720px"></p><h2>Overall Mechanism</h2><ol><li><strong>Closed</strong>: All calls are forwarded. If failures exceed the threshold (e.g., 5), it switches to Open.</li><li><strong>Open</strong>: Calls are blocked, and a <em>fallback</em> is used. After a set time (e.g., 30 seconds), it moves to Half-Open.</li><li><strong>Half-Open</strong>: A test call is made. Success → Closed, Failure → Open.</li></ol><h3>Simple Code Example</h3><p>Here’s a basic implementation in JavaScript:</p><div><pre><code>class CircuitBreaker {
&nbsp; constructor(maxFailures = 5, resetTimeout = 30000) {
&nbsp; &nbsp; this.state = "CLOSED";
&nbsp; &nbsp; this.failureCount = 0;
&nbsp; &nbsp; this.maxFailures = maxFailures;
&nbsp; &nbsp; this.resetTimeout = resetTimeout;
&nbsp; }


&nbsp; async call(service) {
&nbsp; &nbsp; if (this.state === "OPEN") {
&nbsp; &nbsp; &nbsp; if (Date.now() &gt; this.resetTime) {
&nbsp; &nbsp; &nbsp; &nbsp; this.state = "HALF_OPEN";
&nbsp; &nbsp; &nbsp; } else {
&nbsp; &nbsp; &nbsp; &nbsp; return "Fallback: Service unavailable";
&nbsp; &nbsp; &nbsp; }
&nbsp; &nbsp; }


&nbsp; &nbsp; try {
&nbsp; &nbsp; &nbsp; const result = await service();
&nbsp; &nbsp; &nbsp; if (this.state === "HALF_OPEN") {
&nbsp; &nbsp; &nbsp; &nbsp; this.state = "CLOSED";
&nbsp; &nbsp; &nbsp; &nbsp; this.failureCount = 0;
&nbsp; &nbsp; &nbsp; }
&nbsp; &nbsp; &nbsp; return result;
&nbsp; &nbsp; } catch (error) {
&nbsp; &nbsp; &nbsp; this.failureCount++;
&nbsp; &nbsp; &nbsp; if (this.failureCount &gt;= this.maxFailures) {
&nbsp; &nbsp; &nbsp; &nbsp; this.state = "OPEN";
&nbsp; &nbsp; &nbsp; &nbsp; this.resetTime = Date.now() + this.resetTimeout;
&nbsp; &nbsp; &nbsp; }
&nbsp; &nbsp; &nbsp; return "Fallback: Service unavailable";
&nbsp; &nbsp; }
&nbsp; }
}


// Example usage
const breaker = new CircuitBreaker();
const fakeService = () =&gt; Math.random() &gt; 0.5 ? "Success" : Promise.reject("Error");
breaker.call(fakeService).then(console.log);
</code></pre></div><h2>Circuit Breakers in Serverless</h2><p>In a <em>serverless</em> environment (e.g., AWS Lambda), <em>circuit breakers</em> are still valuable, but their stateless nature poses challenges. The state must be stored externally, such as in DynamoDB.</p><h3>Example in AWS Lambda</h3><div><pre><code>const AWS = require('aws-sdk');
const dynamodb = new AWS.DynamoDB.DocumentClient();


async function handler(event) {
&nbsp; const serviceName = "ExternalAPI";
&nbsp; const state = await dynamodb.get({
&nbsp; &nbsp; TableName: "CircuitBreakerState",
&nbsp; &nbsp; Key: { Service: serviceName }
&nbsp; }).promise();


&nbsp; if (state.Item?.State === "OPEN" &amp;&amp; Date.now() &lt; state.Item.ResetTime) {
&nbsp; &nbsp; return { statusCode: 503, body: "Service unavailable" };
&nbsp; }


&nbsp; try {
&nbsp; &nbsp; const response = await callExternalAPI();
&nbsp; &nbsp; if (state.Item?.State === "HALF_OPEN") {
&nbsp; &nbsp; &nbsp; await dynamodb.update({
&nbsp; &nbsp; &nbsp; &nbsp; TableName: "CircuitBreakerState",
&nbsp; &nbsp; &nbsp; &nbsp; Key: { Service: serviceName },
&nbsp; &nbsp; &nbsp; &nbsp; UpdateExpression: "SET #state = :closed",
&nbsp; &nbsp; &nbsp; &nbsp; ExpressionAttributeNames: { "#state": "State" },
&nbsp; &nbsp; &nbsp; &nbsp; ExpressionAttributeValues: { ":closed": "CLOSED" }
&nbsp; &nbsp; &nbsp; }).promise();
&nbsp; &nbsp; }
&nbsp; &nbsp; return { statusCode: 200, body: response };
&nbsp; } catch (error) {
&nbsp; &nbsp; // Logic to update failure count and switch to Open
&nbsp; &nbsp; return { statusCode: 503, body: "Service unavailable" };
&nbsp; }
}
</code></pre></div><h2>Conclusion</h2><p><em>Circuit breakers</em> are a powerful pattern for building resilient systems, whether on traditional servers or in <em>serverless</em> environments. With the simulations and code above, I hope you’ve gained a clearer understanding of how they work.</p>

Imagine you’re using electricity at home, and a short circuit occurs. The circuit breaker in your electrical panel cuts the power to prevent a fire. In software, the concept is similar: it’s a design pattern that protects your system from repeated failures when calling external services

Understanding Circuit Breakers in Software Engineering: From Traditional to Serverless

<h2>Why Choose Apache Cassandra?</h2><p>Unlike traditional relational databases, Cassandra is optimized for handling large workloads across distributed environments. Here’s why it stands out:</p><ul><li><strong>High Availability</strong>: With no single point of failure, Cassandra ensures continuous uptime.</li><li><strong>Horizontal Scalability</strong>: Easily scale out by adding more nodes, avoiding the limitations of vertical scaling.</li><li><strong>Fault Tolerance</strong>: Data replication across nodes guarantees resilience even in case of hardware failures.</li><li><strong>Optimized for Write Operations</strong>: Handles high-speed writes efficiently while offering reliable read performance.</li><li><strong>Flexible Schema</strong>: Unlike relational databases, Cassandra allows schema evolution without downtime.</li></ul><h2>Key Architecture Components</h2><h3>1. <strong>Nodes, Clusters, and Data Centers</strong></h3><ul><li><strong>Node</strong>: The fundamental unit storing a portion of the data.</li><li><strong>Cluster</strong>: A network of nodes working together as a single system.</li><li><strong>Data Center</strong>: A logical grouping of nodes, often used to enhance redundancy across geographical regions.</li></ul><h3>2. <strong>Partitioning &amp; Token Ring</strong></h3><p>Cassandra distributes data across nodes using a <strong>partitioning strategy</strong>, ensuring efficient load balancing. Each node is assigned a <strong>token range</strong>, and data is evenly distributed in a <strong>ring-based architecture</strong>.</p><h3>3. <strong>Replication &amp; Consistency</strong></h3><p>To ensure data availability and reliability, Cassandra employs <strong>replication</strong>:</p><ul><li><strong>Replication Factor (RF)</strong>: Defines the number of copies of data stored across nodes.</li><li><strong>Consistency Levels</strong>: Controls how many nodes must acknowledge a read/write operation (e.g., ONE, QUORUM, ALL), allowing applications to balance performance and reliability.</li></ul><h3>4. <strong>Storage Engine: Commit Log &amp; SSTables</strong></h3><ul><li><strong>Commit Log</strong>: A write-ahead log that captures every write operation for durability before data is flushed to disk.</li><li><strong>Memtable</strong>: A temporary in-memory data structure where writes are stored before being persisted to SSTables.</li><li><strong>SSTables (Sorted String Tables)</strong>: Immutable, append-only files storing actual data on disk, ensuring efficient retrieval and compaction.</li><li><strong>Compaction</strong>: The process of merging multiple SSTables to optimize read performance and free up disk space.</li></ul><h3>5. <strong>Gossip Protocol &amp; Failure Detection</strong></h3><p>Cassandra nodes communicate using the <strong>Gossip Protocol</strong>, a peer-to-peer mechanism for state-sharing, failure detection, and decentralized management.</p><ul><li>Each node periodically exchanges state information with a subset of other nodes.</li><li>Helps maintain a decentralized and resilient system by enabling automatic failure recovery.</li></ul><h3>6. <strong>Read &amp; Write Path in Cassandra</strong></h3><h4><strong>Write Path:</strong></h4><ol><li>Data is written to the <strong>Commit Log</strong> for durability.</li><li>The data is then stored in a <strong>Memtable</strong> (in-memory structure).</li><li>Once the Memtable reaches its threshold, data is flushed to <strong>SSTables</strong> on disk.</li><li>Periodic <strong>compaction</strong> optimizes storage by merging SSTables.</li></ol><h4><strong>Read Path:</strong></h4><ol><li>Cassandra checks the <strong>Memtable</strong> for the latest data.</li><li>If not found, it queries <strong>Bloom Filters</strong> to identify relevant SSTables.</li><li>Reads data from SSTables and merges results before returning them to the client.</li></ol><h2>How Data is Stored &amp; Queried</h2><h3><strong>Primary Keys &amp; Partitions</strong></h3><p>Cassandra structures data into <strong>tables</strong>, similar to relational databases, but with more flexibility. Each table relies on a <strong>Primary Key</strong>, which consists of:</p><ul><li><strong>Partition Key</strong>: Determines data distribution across nodes.</li><li><strong>Clustering Key</strong>: Defines the sorting order of data within a partition.</li></ul><h3><strong>Querying with CQL (Cassandra Query Language)</strong></h3><p>Cassandra utilizes CQL, a SQL-like query language tailored for distributed storage.</p><h4>Example Table Creation:</h4><div><pre><code>CREATE TABLE users (
  id UUID PRIMARY KEY,
  name TEXT,
  email TEXT,
  age INT
);
</code></pre></div><p>However, to maintain speed and efficiency, Cassandra does not support SQL-like JOINs and complex ACID transactions.</p><h2>When to Use Cassandra?</h2><h3><strong>Best Use Cases:</strong></h3><ul><li>Applications requiring <strong>high availability</strong> (e.g., messaging apps, IoT data processing, recommendation engines)</li><li>Large-scale <strong>real-time analytics</strong></li><li><strong>Distributed content delivery</strong> systems</li><li><strong>Financial services</strong> handling time-series data</li></ul><h3><strong>Not Ideal For:</strong></h3><ul><li>Complex transactional applications requiring <strong>strict ACID compliance</strong></li><li>Applications needing frequent <strong>JOIN operations</strong> and deep relational modeling</li></ul><h2>Conclusion</h2><p>Apache Cassandra is a powerful NoSQL database designed for organizations that need to manage high-velocity, large-scale data efficiently. Its distributed architecture, fault tolerance, and seamless scalability make it a prime choice for modern applications handling mission-critical workloads. If you're looking for a battle-tested NoSQL solution capable of global-scale operations, Cassandra is worth exploring!</p>

In today's world of big data, scalability and performance are crucial. Apache Cassandra, an open-source NoSQL database, is a top choice for handling large-scale, distributed data. Used by giants like Facebook, Netflix, and Twitter, Cassandra offers high availability, fault tolerance, and seamless scalability. Let’s dive into its architecture and key concepts!

Apache Cassandra: The NoSQL Powerhouse

<h1><strong>The Mistake That Taught Me a Lesson</strong></h1><p>One day, I was cleaning up an old project, removing unused routes and refactoring the backend. There was this one API endpoint—let's call it /user/details—that I thought was no longer in use. Without a second thought, I deleted it and pushed the changes to production.</p><p>A few hours later, I started receiving errors from another service I had built months earlier. This service, which I had completely forgotten about, was still making requests to /user/details. Suddenly, parts of my application were broken, and I had no easy way to recover from it.</p><p>That was the moment I truly understood why API versioning is critical.</p><h2><strong>Why API Versioning Matters</strong></h2><p><strong>1. Prevents Breaking Changes</strong></p><p>When APIs evolve, clients relying on them should not break due to changes. By implementing versioning (e.g., /v1/user/details), I could have introduced a new version while keeping the old one intact for existing consumers.</p><p><strong>2. Maintains Backward Compatibility</strong></p><p>Even if you think an API is no longer needed, there’s a chance some service or third-party client is still using it. Versioning allows developers to deprecate old APIs gradually rather than abruptly removing them.</p><p><strong>3. Gives Users Time to Migrate</strong></p><p>If an API must change, users need time to update their applications. Providing multiple versions (e.g., /v1/, /v2/) ensures a smooth transition.</p><p><strong>4. Helps in Debugging and Maintenance</strong></p><p>When multiple versions exist, issues can be traced more easily. If a bug appears in /v2/ but not in /v1/, it’s easier to identify what changes might have caused it.</p><h2>How to Implement API Versioning</h2><h3>1. <strong>URL Versioning</strong></h3><p>The most common and widely adopted approach to API versioning is using version numbers in the URL.</p><div><pre><code>/v1/users
/v2/users
</code></pre></div><h4>Pros:</h4><ul><li><strong>Easy to understand and implement</strong> – Developers can quickly identify which version is being used.</li><li><strong>Clear distinction between versions</strong> – Each version has its own endpoint, ensuring that changes do not interfere with older versions.</li></ul><h4>Cons:</h4><ul><li><strong>Can lead to bloated URLs</strong> – If too many versions exist, the API can become cluttered.</li><li><strong>Might require modifying routes and maintaining multiple endpoints</strong> – Developers must maintain multiple versions, which can increase complexity over time.</li></ul><h3>2. <strong>Header Versioning</strong></h3><p>Another approach is to use HTTP headers to specify the API version instead of embedding it in the URL.</p><div><pre><code>Accept: application/vnd.myapi.v1+json
</code></pre></div><h4>Pros:</h4><ul><li><strong>Keeps URLs clean</strong> – There’s no need to modify the URL structure, making it aesthetically cleaner.</li><li><strong>Allows more flexibility without changing routes</strong> – Clients can request different versions dynamically using headers.</li></ul><h4>Cons:</h4><ul><li><strong>Requires clients to send custom headers explicitly</strong> – Clients must be aware of the correct headers to use, which adds complexity.</li><li><strong>Might be harder to test and debug compared to URL versioning</strong> – Since versioning is not visible in the URL, debugging and API documentation can be more challenging.</li></ul><h3>3. <strong>Query Parameter Versioning</strong></h3><p>This method involves specifying the API version as a query parameter in the request.</p><div><pre><code>/users?version=1
</code></pre></div><h4>Pros:</h4><ul><li><strong>Simple to implement and does not require changes to routes</strong> – The backend can handle different versions without modifying the API structure.</li><li><strong>Can be easily handled on the backend</strong> – Developers can dynamically parse the version parameter and route requests accordingly.</li></ul><h4>Cons:</h4><ul><li><strong>Can lead to inconsistent API calls if clients forget to include the version</strong> – If a request is made without the version parameter, it may result in unintended behavior.</li><li><strong>May clutter the query string with additional parameters</strong> – This approach can become cumbersome if multiple parameters are needed.</li></ul><h2><strong>Choosing the Right API Versioning Strategy</strong></h2><p>Each of these methods has its strengths and weaknesses, and the best approach depends on the specific needs of your project. If you want a simple and widely understood method, <strong>URL versioning</strong> might be the best choice. If you prefer a cleaner URL structure, <strong>header versioning</strong> could be a better fit. And if you need quick implementation without altering routes, <strong>query parameter versioning</strong> is a viable option.</p><h2><strong>Final Thoughts</strong></h2><p>I learned the hard way that careless API deletions can lead to unexpected failures. If I had implemented proper versioning, I could have safely iterated on my APIs without breaking my own services.</p><p>So, if you're developing APIs—whether for personal projects or production systems—take API versioning seriously. Your future self (and your users) will thank you!</p>

As a developer, I've built countless APIs for my personal projects. Some were experimental, some turned into full-fledged applications, and others were simply abandoned over time. At first, managing these APIs felt simple—if I wasn't using an endpoint anymore, I would just delete it. Why keep something that I no longer need, right?  Well, that mindset came back to bite me.

Why API Versioning is Really Important: A Lesson from My Own Mistake

<h2><strong>How Does Terraform Work?</strong></h2><h3><strong>1. Declarative Configuration</strong></h3><p>Terraform uses a special language called <strong>HashiCorp Configuration Language (HCL)</strong>. Instead of writing step-by-step instructions like in traditional programming, you just <strong>describe what you want</strong> (e.g., "I need a virtual machine with 2 CPUs and 4GB RAM"), and Terraform figures out how to make it happen.</p><h3><strong>2. Configuration Interpretation</strong></h3><p>When you run a Terraform command, it reads your configuration files and <strong>understands what infrastructure you want to create or update</strong>.</p><h3><strong>3. Interaction with Cloud APIs</strong></h3><p>Terraform then communicates with cloud providers like <strong>Google Cloud, AWS, or Azure</strong> by sending <strong>API requests</strong>. This tells the cloud provider to create, update, or delete the resources you defined.</p><h3><strong>4. Execution by Cloud Providers</strong></h3><p>The cloud provider takes Terraform’s instructions and <strong>builds your infrastructure</strong>—creating things like virtual machines, networks, and storage based on your configuration.</p><h3><strong>5. State Management</strong></h3><p>Terraform keeps track of everything it has created in a <strong>state file</strong>. This file helps Terraform know what’s already there, so it only makes <strong>necessary changes</strong> when you update your configuration.</p><h3><strong>Why Use Terraform?</strong></h3><p>Terraform makes infrastructure management <strong>simpler, repeatable, and error-free</strong>. Instead of manually clicking around in cloud dashboards, you can <strong>automate everything</strong> with a few lines of code. This saves time and reduces mistakes.</p><p>Terraform enables you to safely and predictably create, change, and improve infrastructure. It is an open-source tool that codifies APIs into declarative configuration files that can be shared among team members, treated as code, edited, reviewed, and versioned.</p><p>In this lab, you create a Terraform configuration with a module to automate the deployment of Google Cloud infrastructure.</p><h2><strong>Task 1: Setting Up Terraform and Cloud Shell</strong></h2><h3><strong>Installing Terraform</strong></h3><p>Terraform is pre-installed in Cloud Shell. Verify the installed version.</p><p>1 Open <strong>Google Cloud Console</strong> and click <strong>Activate Cloud Shell</strong>.</p><p>2 If prompted, click <strong>Continue</strong>.</p><p>3 Run the following command to check the Terraform version:</p><div><pre><code>terraform --version
</code></pre></div><p><strong>- Expected output:</strong></p><div><pre><code>Terraform v1.3.3
</code></pre></div><blockquote><strong>Note:</strong> These lab instructions work with Terraform v1.3.3 and later.</blockquote><p>4 Create a directory for Terraform configurations:</p><div><pre><code>mkdir tfinfra
</code></pre></div><p>5 Open <strong>Cloud Shell Editor</strong> and navigate to the <strong><em>tfinfra </em></strong>folder.</p><h3><strong>Initializing Terraform</strong></h3><p>Terraform uses plugins to support various cloud providers. Initialize Terraform by setting Google as the provider.</p><p>1 Create a new file named <strong>provider.tf</strong> tfinfra folder.</p><p>2 Add the following configuration:</p><div><pre><code>provider "google" {}
</code></pre></div><p>3 Save the file.</p><p>4 Run the Terraform initialization command:</p><div><pre><code>cd tfinfra
terraform init
</code></pre></div><p><strong>- Expected output:</strong></p><div><pre><code>provider.google: version = "~&gt; 4.43.0"
Terraform has been successfully initialized!
</code></pre></div><h2><strong>Task 2: Creating mynetwork and Its Resources</strong></h2><h3><strong>Configuring mynetwork</strong></h3><p>1 Create a new file named <strong>mynetwork.tf</strong> inside tfinfra.</p><p>2 Add the following configuration:</p><div><pre><code>resource "google_compute_network" "mynetwork" {
  name                    = "mynetwork"
  auto_create_subnetworks = true
}
</code></pre></div><p>3 Save the file.</p><h3><strong>Configuring Firewall Rules</strong></h3><p>4 Add the following firewall rules to mynetwork,tf:</p><div><pre><code>resource "google_compute_firewall" "mynetwork-allow-http-ssh-rdp-icmp" {
  name    = "mynetwork-allow-http-ssh-rdp-icmp"
  network = google_compute_network.mynetwork.self_link

  allow {
    protocol = "tcp"
    ports    = ["22", "80", "3389"]
  }
  allow {
    protocol = "icmp"
  }
  source_ranges = ["0.0.0.0/0"]
}
</code></pre></div><p>5 Save the file.</p><h3><strong>Configuring VM Instance</strong></h3><p>1 Create a new folder named <strong>instance</strong> inside tfinfra.</p><p>2 Create a new file <strong>main.tf</strong> inside the instance folder.</p><p>3 Add the following basic configuration:</p><div><pre><code>resource "google_compute_instance" "vm_instance" {
  name         = "my-vm-instance"
  machine_type = "e2-medium"
  zone         = "us-central1-a"

  boot_disk {
    initialize_params {
      image = "debian-cloud/debian-10"
    }
  }
  network_interface {
    network = google_compute_network.mynetwork.self_link
  }
}
</code></pre></div><p>4 Save the file.</p><p>To rewrite the Terraform configuration files to a canonical format and style, run the following command:</p><div><pre><code>terraform fmt
</code></pre></div><p>To initialize Terraform, run the following command:</p><div><pre><code>terraform init
</code></pre></div><p><strong>Expected output:</strong></p><div><pre><code>...
* provider.google: version = "~&gt; 4.43.0"

Terraform has been successfully initialized!
</code></pre></div><h2><strong>Conclusion</strong></h2><p>You have successfully set up Terraform in Cloud Shell and created configurations to deploy Google Cloud infrastructure, including a VPC network, firewall rules, and VM instances. Terraform’s execution workflow ensures smooth infrastructure deployment with minimal manual intervention. This setup can be expanded with additional configurations and modules to efficiently automate more complex infrastructure deployments.</p><p>Happy learning and coding with Terraform!</p>

Manually managing cloud infrastructure can be time-consuming and error-prone. Terraform changes the game by allowing you to define infrastructure as code, making deployment faster, scalable, and repeatable. With Terraform, you can automate cloud resource creation, track changes, and collaborate effortlessly.

Terraform Labs: Automating Google Cloud Infrastructure Deployment

<h1><strong>Why Use Jenkins for CI/CD?</strong></h1><p>Jenkins is not just a tool; it's a culture shifter. With its unparalleled flexibility, thousands of plugins, and vibrant community, Jenkins transforms how teams approach Continuous Integration and Continuous Delivery (CI/CD). Whether you’re a small startup racing to push your MVP or a large enterprise managing complex workflows, Jenkins empowers you to automate repetitive tasks, reduce errors, and accelerate delivery pipelines.</p><p>But why Jenkins? It’s open-source, highly customizable, and scales effortlessly with your team’s growing needs. It’s the bridge between developers and operations teams, breaking down silos and fostering collaboration in ways you’ve never experienced before.</p><p>In this labs, we’ll explore the magic of Jenkins and why it’s the go-to choice for CI/CD pipelines. Ready to revolutionize your development workflow? Let’s dive in!</p><h1><strong>Setting Up CI/CD in Jenkins</strong></h1><p>To start setting up CI/CD with Jenkins, the first step is to prepare your environment by installing Java. Jenkins requires Java to run, so you need to install OpenJDK 17. Update your system packages and install Java by running the following commands:</p><div><pre><code>sudo apt update
sudo apt install fontconfig openjdk-17-jre
java -version
</code></pre></div><p>After installation, you can verify the version of Java to ensure everything is properly set up. The output should display the installed version, such as openJDK version 17. With Java in place, your system is now ready to host Jenkins.</p><p>Next, install Jenkins using its official package repository to ensure you're getting the latest stable release. First, add Jenkins' repository key and configuration to your system, update the package list, and install Jenkins by running these commands:</p><div><pre><code>sudo wget -O /usr/share/keyrings/jenkins-keyring.asc \
  https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key
echo "deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc]" \
  https://pkg.jenkins.io/debian-stable binary/ | sudo tee \
  /etc/apt/sources.list.d/jenkins.list &gt; /dev/null
sudo apt-get update
sudo apt-get install jenkins
</code></pre></div><p>Once Jenkins is installed, you need to enable and start the Jenkins service. This ensures Jenkins runs immediately and starts automatically whenever the system boots. Use the following commands to enable and start Jenkins:</p><div><pre><code>sudo systemctl enable jenkins
sudo systemctl start jenkins
sudo systemctl status jenkins
</code></pre></div><p>After starting the service, you can access Jenkins through your browser. Open http://[VM_EXTERNAL_IP]:8080&nbsp;and log in using the initial admin password. To retrieve this password, run:</p><div><pre><code>sudo cat /var/lib/jenkins/secrets/initialAdminPassword
</code></pre></div><p>Enter the password on the setup screen and follow the guided setup wizard. This process will include installing recommended plugins, such as Git, Pipeline, Blue Ocean, and GitHub, which are essential for building your CI/CD pipeline.</p><p>In addition to plugins, configure credentials for accessing your source code repository. Navigate to <strong>Manage Jenkins &gt; Credentials</strong>, and add secure credentials for platforms like GitHub or GitLab. This setup ensures your pipeline can pull code from your repositories securely.</p><p>With Jenkins configured, you’re ready to create your first pipeline. The pipeline script can define all stages of your CI/CD process, such as fetching code, running tests, and deploying to production.</p><h1><strong>Setting Up Git Credentials for Private Repositories</strong></h1><p>If your project is hosted on a private Git platform, you’ll need to provide secure credentials for Jenkins to access the repository. For GitHub users, this involves creating a Personal Access Token (PAT) and adding it to Jenkins' credentials. This ensures seamless integration between Jenkins and your repository without compromising security.</p><h2><strong>Creating a Personal Access Token (PAT) on GitHub</strong></h2><p>To create a PAT in GitHub:</p><p>1. Go to your GitHub account settings.</p><p>2. Navigate to Developer settings &gt; Personal Access Tokens &gt; Tokens (classic).</p><p>3. Click Generate new token and specify the required permissions. For most CI/CD setups, select scopes like repo (for repository access) and workflow (if managing GitHub Actions).</p><p>4. Generate the token and copy it. Make sure to store it securely as you won’t be able to view it again.</p><h2>Configuring Jenkins with GitHub Credentials</h2><p>Once you’ve created your PAT, follow these steps to add it to Jenkins:</p><p>1. Log in to the Jenkins Dashboard.</p><p>2. Navigate to Manage Jenkins &gt; Manage Credentials.</p><p>3. Under (global) credentials, click Add Credentials.</p><p>4. In the Kind dropdown, select Username with password.</p><p>Username: Enter your GitHub username (e.g., username).</p><p>Password: Paste the PAT you just created.</p><p>5. Give the credential a recognizable ID (e.g., github-creds) and click OK.</p><h1>Writing Your First Jenkins Pipeline</h1><p>With Jenkins configured, you’re ready to create your first pipeline. The pipeline script defines all stages of your CI/CD process, including fetching code, running tests, and deploying the application. Let’s walk through an example pipeline written using Jenkins' declarative syntax, which simplifies the process and ensures better readability.</p><h4>Understanding the Pipeline Script</h4><p>Below is an example pipeline that automates three key stages of the CI/CD process:</p><p><strong>1. Fetching the code from GitHub</strong> using credentials for a secure connection.</p><p><strong>2. Running basic testing commands</strong> to verify the environment setup.</p><p><strong>3. Deploying the application</strong> to a server using SSH for remote commands</p><div><pre><code>pipeline {
&nbsp; &nbsp; agent any
&nbsp; &nbsp;&nbsp;
&nbsp; &nbsp; environment {
&nbsp; &nbsp; &nbsp; &nbsp; GITHUB_CREDENTIALS = 'caea020d-a24e-4305-bdc2-d7e51d1c8171'&nbsp; // ID of the GitHub credentials in Jenkins
&nbsp; &nbsp; }
&nbsp; &nbsp;&nbsp;
&nbsp; &nbsp; stages {
&nbsp; &nbsp; &nbsp; &nbsp; stage('Git Checkout') {
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; steps {
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; script {
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; // Cloning the GitHub repository using the provided credentials
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; git credentialsId: "${GITHUB_CREDENTIALS}", url: 'https://github.com/Barbarpotato/API-Registry.git', branch: 'main'
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }
&nbsp; &nbsp; &nbsp; &nbsp; }
&nbsp; &nbsp; &nbsp; &nbsp;&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; stage('Run Testing Commands') {
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; steps {
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; sh 'hostname'&nbsp; // Outputs the hostname of the Jenkins agent
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; sh 'pwd'&nbsp; &nbsp; &nbsp; &nbsp;// Displays the current working directory in the agent's workspace
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }
&nbsp; &nbsp; &nbsp; &nbsp; }
&nbsp; &nbsp; &nbsp; &nbsp;&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; stage('Deploy to Server') {
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; steps {
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; // Using SSH to deploy the application to the target server
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; sshagent(['ssh-key-gateway']) {
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; sh '''
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ssh -o StrictHostKeyChecking=no darmawanjr88@34.101.205.217 &lt;&lt; EOF
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; cd API-Registry
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; git pull origin main
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; pm2 restart all
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; exit
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; EOF
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; '''
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }
&nbsp; &nbsp; &nbsp; &nbsp; }
&nbsp; &nbsp; }
}
</code></pre></div><p>The provided pipeline script is a clear example of how Jenkins orchestrates the CI/CD process, broken into multiple stages for ease of management. Let's break it down section by section and explain how each part works in a simplified, interactive way.</p><h3>Pipeline Declaration</h3><p>At the heart of any Jenkins pipeline is the pipeline block, which defines the entire process. Inside this block, the agent any directive tells Jenkins to run the pipeline on any available agent, whether it's a master or a worker node. This flexibility is useful when you have multiple agents configured and don’t want to restrict the execution to a specific one.</p><h3>Environment Variables</h3><p>Next, we have the environment block. This is where we can define variables that are reused across the pipeline. In this case, the variable GITHUB_CREDENTIALS holds the ID of the GitHub credentials that Jenkins uses to securely access the private repository. By defining it here, you ensure it’s easily reusable without hardcoding it into every step. Think of it as a centralized way to manage sensitive data like tokens and credentials, making the pipeline both secure and maintainable.</p><h3>Stages</h3><p>The pipeline is broken into logical steps called "stages," each representing a part of the CI/CD workflow.</p><p>1. Git Checkout</p><p>In the first stage, Jenkins clones the GitHub repository using the git step. The credentialsId points to the pre-configured GitHub credentials stored in Jenkins. This ensures secure and seamless access to the private repository without exposing sensitive information. This step lays the foundation for the entire pipeline, as it fetches the code that the remaining stages will process.</p><p>2. Run Testing Commands</p><p>This stage is simple but powerful. It runs shell commands such as hostname and pwd, which output the hostname of the Jenkins agent and the current working directory, respectively. While these commands are placeholders here, you can replace them with actual test scripts. For instance, if you’re running unit tests, you could include a command like npm test or pytest. The purpose of this stage is to ensure the environment is configured correctly and ready for further operations.</p><p>3. Deploy to Server</p><p>This stage demonstrates how Jenkins can deploy your application to a production or staging server. Using the sshagent block, Jenkins securely connects to the target server via SSH. It then pulls the latest changes from the repository and restarts the application using PM2, a popular process manager for Node.js. This setup ensures that the application is always up-to-date with the latest code changes, and the restart command ensures a smooth rollout of updates.</p><p>Continuing from the previous explanation of setting up a Jenkins pipeline, the next step to truly automate the CI/CD process is to set up a <strong>push trigger</strong> using webhooks. This ensures that every time you push changes to your GitHub repository, Jenkins automatically triggers the pipeline, saving you from the hassle of manually starting the build.</p><p>Let’s explore how to set up a webhook-based trigger between GitHub and Jenkins in an intuitive and straightforward way.</p><h3><strong>What Are Webhooks?</strong></h3><p>Think of webhooks as a way for GitHub to "talk" to Jenkins. Whenever you push code to your repository, GitHub sends a signal (HTTP POST request) to Jenkins, telling it to start the pipeline. This creates an automated, real-time link between your code changes and the build process.</p><h3><strong>Setting Up Push Trigger (Webhook)</strong></h3><h4><strong>First, Enable GitHub Integration in Jenkins</strong></h4><p>Before setting up the webhook, you need to make sure Jenkins can communicate with GitHub. To do this, open Jenkins and go to <strong>Manage Jenkins</strong> &gt; <strong>Manage Plugins</strong>. From there, search for <strong>GitHub</strong> plugins and install them. This will allow Jenkins to recognize GitHub as a source and receive notifications from it.</p><h4><strong>Configure Jenkins to Listen for Webhooks</strong></h4><p>Next, open the Jenkins pipeline job you want to configure. In the job settings, go to <strong>Build Triggers</strong> and enable the option <strong>GitHub hook trigger for GITScm polling</strong>. This step is important because it tells Jenkins to "listen" for any push events from GitHub, ready to trigger the pipeline whenever changes are detected.</p><h4><strong>Set Up a Webhook in GitHub</strong></h4><p>Now, head over to your GitHub repository. Inside the <strong>Settings</strong> tab, navigate to <strong>Webhooks</strong> and click on <strong>Add webhook</strong>. You'll need to provide Jenkins with a specific endpoint where it can receive notifications from GitHub. The URL format is as follows:</p><div><pre><code>http://&lt;JENKINS_URL&gt;/github-webhook/
</code></pre></div><p>Replace <strong><em>JENKINS_URL </em></strong>with your Jenkins server’s address. Choose <strong>application/json</strong> as the content type, and make sure the <strong>Push events</strong> option is selected. This ensures that the webhook triggers every time you push changes.</p><h4><strong>Verify the Webhook</strong></h4><p>Once everything is set up, push a commit to your GitHub repository. Then, return to Jenkins and check if the pipeline starts running automatically. If the job kicks off, the webhook is working properly.</p><h3><strong>How It All Comes Together</strong></h3><p>When you push changes to your repository, GitHub sends a webhook to Jenkins. Jenkins then triggers the pipeline to fetch the latest code, run tests, and deploy to your server. This creates a seamless CI/CD process, where every change is automatically tested and deployed.</p><h1><strong>Conclusion</strong></h1><p>In this Labs, we’ve taken a deep dive into how Jenkins can supercharge your CI/CD workflows, making it an essential tool for automating your development lifecycle. We started by setting up Jenkins, installing necessary dependencies like Java, and getting the Jenkins service up and running.</p><p>From there, we explored how to configure Jenkins to work with your GitHub repository, including managing credentials securely. With Jenkins set up and connected to GitHub, we moved on to creating a simple declarative pipeline, allowing Jenkins to automatically fetch the latest code, run tests, and deploy to your server.</p><p>We then enhanced the process by explaining how to set up push triggers using GitHub webhooks. With webhooks in place, Jenkins is able to automatically start the pipeline whenever new code is pushed to the repository, eliminating the need for manual intervention and ensuring continuous integration.</p><p>Through these steps, we’ve created a fully automated CI/CD pipeline that reacts to changes in your code, testing it and deploying it seamlessly. This not only saves time but also minimizes errors, providing faster and more reliable software delivery.</p><p>By mastering Jenkins, you’re empowering yourself to automate complex workflows, improve collaboration, and focus on building great software without worrying about the manual process of integration and deployment.</p>

In the fast-paced world of modern software development, delivering high-quality applications quickly is no longer optional—it's essential. This is where Jenkins steps in as a game-changer. Imagine having a virtual assistant that tirelessly builds, tests, and deploys your code, ensuring every update you make reaches production seamlessly.

Jenkins Unleashed: Transforming Your CI/CD Workflow for Lightning-Fast Delivery

In the ever-evolving world of software architecture, building a robust and scalable system is key to meeting the demands of modern applications. Recently, I had the opportunity to explore a powerful combination of technologies, starting with gRPC, translating it into HTTP/1.1, and finally deploying the system to a Kubernetes cluster. In this blog, I will take you through the journey, share the challenges I encountered, and explain why each of these steps is important for modern software systems.

Building a Robust Microservices Architecture: From gRPC to Kubernetes

<p>By the end of this guide, you'll have the skills to:</p><p>- Store video chunks securely in AWS S3.</p><p>- Stream videos dynamically to users using HLS.</p><p>- Enable users to upload videos, automatically process them into HLS chunks, and store them in S3.</p><p><strong>Ready to get started? Let's dive in!</strong></p><h1><strong>Step 1: Storing Video Chunks in AWS S3</strong></h1><p>To stream videos, we first need to upload .ts (MPEG transport stream) files and a .m3u8 playlist to an AWS S3 bucket. Follow these steps to upload your files:</p><h3>Why AWS S3?</h3><p>AWS S3 provides a scalable, reliable, and secure storage solution, making it perfect for handling large video files. Using the AWS CLI simplifies the upload process and integrates easily into automation pipelines.</p><h3>Step-by-Step Instructions:</h3><p>1. Open your terminal.</p><p>2. Run the following commands to upload the video chunks and playlist:</p><div><pre><code>aws s3 cp output0.ts s3://your-bucket-name/path-to-folder/
aws s3 cp playlist.m3u8 s3://your-bucket-name/path-to-folder/
</code></pre></div><p>While we’re using the AWS CLI here, you could also use the AWS Management Console or SDKs (e.g., AWS SDK for JavaScript) if you prefer a graphical interface or code-based interaction.</p><p>Once uploaded, your files will be securely stored and accessible for streaming. Let’s move to setting up the backend!</p><h1><strong>Step 2: Streaming the Video with Express.js</strong></h1><p>Now that our video chunks are in S3, we need a backend to fetch and serve them to the client. We’ll use Node.js with Express.js to handle this.</p><h3>Backend Setup:</h3><p>1. Create a new Node.js project and install the necessary packages:</p><div><pre><code>npm init -y
npm install express @aws-sdk/client-s3
</code></pre></div><p>2. Add the following code to your server.js file: </p><div><pre><code>import express from 'express';
import { S3Client, GetObjectCommand } from '@aws-sdk/client-s3';
import stream from 'stream';

const app = express();
const s3 = new S3Client({ region: 'your-region' });

// Endpoint to serve the HLS playlist
app.get('/play/playlist.m3u8', async (req, res) =&gt; {
&nbsp;&nbsp;try {
&nbsp;&nbsp;&nbsp;&nbsp;const command = new GetObjectCommand({
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Bucket: 'your-bucket-name',
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Key: 'path-to-folder/playlist.m3u8',
&nbsp;&nbsp;&nbsp;&nbsp;});
&nbsp;&nbsp;&nbsp;&nbsp;const s3Response = await s3.send(command);
&nbsp;&nbsp;&nbsp;&nbsp;s3Response.Body.pipe(res);
&nbsp;&nbsp;} catch (err) {
&nbsp;&nbsp;&nbsp;&nbsp;console.error('Error fetching playlist:', err);
&nbsp;&nbsp;&nbsp;&nbsp;res.status(500).send('Error fetching playlist');
&nbsp;&nbsp;}
});

// Endpoint to serve video chunks
app.get('/play/video/:segment', async (req, res) =&gt; {
&nbsp;&nbsp;try {
&nbsp;&nbsp;&nbsp;&nbsp;const command = new GetObjectCommand({
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Bucket: 'your-bucket-name',
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Key: `path-to-folder/${req.params.segment}`,
&nbsp;&nbsp;&nbsp;&nbsp;});
&nbsp;&nbsp;&nbsp;&nbsp;const s3Response = await s3.send(command);
&nbsp;&nbsp;&nbsp;&nbsp;s3Response.Body.pipe(res);
&nbsp;&nbsp;} catch (err) {
&nbsp;&nbsp;&nbsp;&nbsp;console.error('Error fetching video segment:', err);
&nbsp;&nbsp;&nbsp;&nbsp;res.status(500).send('Error fetching video segment');
&nbsp;&nbsp;}
});

const PORT = 3000;
app.listen(PORT, () =&gt; {
&nbsp;&nbsp;console.log(`Server is running on http://localhost:${PORT}`);
});
</code></pre></div><h3>What’s Happening Here?</h3><p><strong>- Playlist Endpoint</strong>: Fetches and streams the .m3u8 file from S3.</p><p><strong>- Video Chunks Endpoint</strong>: Dynamically fetches .ts chunks based on the client’s request.</p><p>Once the backend is running, we can serve video content to the client. Let’s bring it all together on the frontend.</p><h1><strong>Step 3: Playing the Video on the Client-Side</strong></h1><p>To play the HLS video stream in the browser, we’ll use the <code>&lt;video&gt;</code> HTML tag and the HLS.js library for compatibility with all modern browsers.</p><h3>Implementation:</h3><p>1. Create an index.html file with the following content:</p><div><pre><code>&lt;!DOCTYPE html&gt;
&lt;html lang="en"&gt;
&lt;head&gt;
&nbsp;&lt;meta charset="UTF-8"&gt;
&nbsp;&lt;meta name="viewport" content="width=device-width, initial-scale=1.0"&gt;
&nbsp;&lt;title&gt;Video Streaming&lt;/title&gt;
&nbsp;&lt;script src="https://cdn.jsdelivr.net/npm/hls.js@latest"&gt;&lt;/script&gt;
&lt;/head&gt;
&lt;body&gt;
&nbsp;&lt;h1&gt;Video Streaming Example&lt;/h1&gt;
&nbsp;&lt;video id="videoPlayer" controls width="720" autoplay&gt;&lt;/video&gt;

&nbsp;&lt;script&gt;
&nbsp;&nbsp;const video = document.getElementById('videoPlayer');
&nbsp;&nbsp;const videoSrc = 'http://localhost:3000/play/playlist.m3u8';

&nbsp;&nbsp;if (Hls.isSupported()) {
&nbsp;&nbsp;&nbsp;const hls = new Hls();
&nbsp;&nbsp;&nbsp;hls.loadSource(videoSrc);
&nbsp;&nbsp;&nbsp;hls.attachMedia(video);
&nbsp;&nbsp;} else if (video.canPlayType('application/vnd.apple.mpegurl')) {
&nbsp;&nbsp;&nbsp;video.src = videoSrc;
&nbsp;&nbsp;} else {
&nbsp;&nbsp;&nbsp;console.error('This browser does not support HLS streaming');
&nbsp;&nbsp;}
&nbsp;&lt;/script&gt;
&lt;/body&gt;
&lt;/html&gt;
</code></pre></div><h3>Key Points:</h3><p>- The browser fetches the <code>.m3u8</code> playlist, which directs it to download the <code>.ts</code> chunks for smooth playback.</p><p>- The HLS.js library ensures compatibility with browsers that don’t natively support HLS.</p><p>Now you have a working video player! Let’s take it a step further by allowing users to upload videos.</p><h1><strong>Step 4: Allowing Users to Upload Videos</strong></h1><p>What if users want to upload their own videos? We can process those videos into HLS chunks and store them in S3.</p><h3>Backend for Uploading and Processing:</h3><p>1. Install the required packages:</p><div><pre><code>npm install multer fluent-ffmpeg
</code></pre></div><p>2. Update your server.js file with the following:</p><div><pre><code>import multer from 'multer';
import ffmpeg from 'fluent-ffmpeg';
import { PutObjectCommand } from '@aws-sdk/client-s3';
import fs from 'fs';

const upload = multer({ dest: 'uploads/' });

app.post('/upload', upload.single('video'), (req, res) =&gt; {
&nbsp;&nbsp;const inputPath = req.file.path;
&nbsp;&nbsp;const outputPath = 'processed/';

&nbsp;&nbsp;ffmpeg(inputPath)
&nbsp;&nbsp;&nbsp;&nbsp;.output(`${outputPath}playlist.m3u8`)
&nbsp;&nbsp;&nbsp;&nbsp;.outputOptions([
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'-hls_time 10',
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'-hls_list_size 0',
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'-f hls',
&nbsp;&nbsp;&nbsp;&nbsp;])
&nbsp;&nbsp;&nbsp;&nbsp;.on('end', async () =&gt; {
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;try {
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;// Upload .m3u8 and .ts files to S3
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;const filesToUpload = fs.readdirSync(outputPath);
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;for (const file of filesToUpload) {
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;const command = new PutObjectCommand({
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Bucket: 'your-bucket-name',
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Key: `path-to-folder/${file}`,
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Body: fs.createReadStream(`${outputPath}${file}`),
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;});
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;await s3.send(command);
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;res.send('Video uploaded and processed successfully!');
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;} catch (err) {
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;console.error('Error uploading to S3:', err);
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;res.status(500).send('Error uploading video.');
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}
&nbsp;&nbsp;&nbsp;&nbsp;})
&nbsp;&nbsp;&nbsp;&nbsp;.on('error', (err) =&gt; {
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;console.error('Error processing video:', err);
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;res.status(500).send('Error processing video.');
&nbsp;&nbsp;&nbsp;&nbsp;})
&nbsp;&nbsp;&nbsp;&nbsp;.run();
});
</code></pre></div><h3>Workflow:</h3><p>1. Users upload a video via the /upload endpoint.</p><p>2. FFmpeg processes the video into HLS chunks.</p><p>3. The processed files are automatically uploaded to S3.</p><h1><strong>Final Thoughts</strong></h1><p>Congratulations! You’ve built a fully functional video streaming platform. Here's a recap of what we accomplished:</p><p><strong>- Secure Storage</strong>: Used AWS S3 for reliable and scalable storage.</p><p><strong>- Seamless Streaming</strong>: Enabled dynamic streaming with HLS.</p><p><strong>- User Uploads</strong>: Automated video processing and storage with FFmpeg and Node.js.</p><p>This architecture is scalable and can serve as the foundation</p>

Ever wondered how your favorite streaming platforms deliver smooth, high-quality videos? Streaming video content is a cornerstone of modern web applications. Let’s explore how to build a video streaming service step by step.

Building a Video Streaming Platform with AWS S3, HLS, and Node.js

<h1><strong>Why You Need Load Balancing</strong></h1><p>Imagine you’re throwing a huge party, and your house is the server. If everyone shows up at once, there’s no room to move, and things get crowded. But if you have a few people guiding guests into different rooms, everyone gets a chance to enjoy the party without overcrowding any single room. This is <strong>Load Balancing</strong>!</p><p>Load balancing is like your traffic manager: it takes the incoming website traffic and evenly distributes it across multiple servers. The result?</p><ul><li><strong>Better Performance</strong>: No server is overloaded.</li><li><strong>More Reliability</strong>: If one server crashes, the others keep the party going.</li><li><strong>Easy Scalability</strong>: As your site grows, you can add more servers to handle more traffic without a hitch.</li></ul><p>In this guide, we’re going to set up a simple load balancing system on AWS using <strong>EC2 instances</strong> and <strong>Nginx</strong> (a popular web server). We’ll skip the fancy AWS Elastic Load Balancer (ELB) for now and do it ourselves. ?</p><h1><strong>The Load Balancing Setup in a Nutshell</strong></h1><h1><img src="https://firebasestorage.googleapis.com/v0/b/personal-blog-darmajr.appspot.com/o/blog-content%2Fload_balancing_structure.png?alt=media&amp;token=92732226-d927-43d0-9d9e-3d82fed692f1" alt="load balancing workflow example for demo" width="720px"></h1><p>In a typical web application, you want to ensure that traffic is efficiently managed across your infrastructure. If all users are directed to a single server, that server might get overloaded, causing slow response times or crashes. This is where <strong>load balancing</strong> comes into play.</p><p>Let’s break down our setup:</p><p><strong>- 1 Load Balancer</strong>: This is like the traffic cop for your web traffic. When users visit your website, their requests first hit the load balancer. The load balancer’s job is to direct these requests to the available backend servers.</p><p><strong>- 2 Server Instances</strong>: These are your backend servers that actually process the users’ requests and return content. The load balancer will send incoming requests to either Server 1 or Server 2, ensuring that no single server is overwhelmed. If one server goes down, the load balancer automatically redirects traffic to the other one, ensuring high availability.</p><h1><strong>What You’ll Need</strong></h1><p>Before we start, make sure you have:</p><p><strong>1. Three EC2 instances</strong> on AWS: One for the load balancer and two for the backend servers.</p><p><strong>2. Nginx installed</strong> on all the instances. It’ll handle the distribution of traffic to the backend servers.</p><h1><strong>Let’s Get Started! Step by Step</strong></h1><h3><strong>Step 1: Launch EC2 Instances in AWS</strong></h3><p>You’ll need three EC2 instances:</p><ul><li><strong>Server 1 (Backend)</strong>: This server will handle part of the traffic.</li><li><strong>Server 2 (Backend)</strong>: Another server to handle a different part of the traffic.</li><li><strong>Load Balancer</strong>: This instance will distribute the traffic between the two backend servers.</li></ul><p>All servers should be in the same <strong>VPC</strong> (Virtual Private Cloud) for easy communication.</p><h3><strong>Step 2: Install Nginx on Each EC2 Instance</strong></h3><p>Once your EC2 instances are running, it’s time to install Nginx. You can do that by connecting to each instance and running:</p><div><pre><code>sudo apt update 
sudo apt install nginx
</code></pre></div><h3><strong>Step 3: Configure Nginx on the Load Balancer</strong></h3><p>Now comes the fun part — setting up Nginx on the <strong>Load Balancer</strong>.</p><p><strong>1. Access your load balancer instance</strong> and open the Nginx configuration file:</p><div><pre><code>sudo nano /etc/nginx/nginx.conf
</code></pre></div><p><strong>2. Add the following configuration</strong> inside the http block:</p><div><pre><code>upstream backend_servers {
&nbsp;&nbsp;server 172.31.27.145;&nbsp;# Private IP of Server 1
&nbsp;&nbsp;server 172.31.27.91;&nbsp;&nbsp;# Private IP of Server 2
}

server {
&nbsp;&nbsp;listen 80;
&nbsp;&nbsp;server_name &lt;load-balancer-public-IP&gt;;

&nbsp;&nbsp;location / {
&nbsp;&nbsp;&nbsp;&nbsp;proxy_pass http://backend_servers;
&nbsp;&nbsp;}
}
</code></pre></div><p>This is telling Nginx, “Hey, I’ve got two backend servers here (with private IPs), and I want to send all incoming traffic to them.” The load balancer will automatically send traffic to either Server 1 or Server 2.</p><p><strong>Check if the configuration is correct</strong>:</p><div><pre><code>sudo nginx -t
</code></pre></div><p><strong>Reload Nginx</strong> to apply the new settings:</p><div><pre><code>sudo systemctl reload nginx
</code></pre></div><h3><strong>Step 4: Test Your Backend Servers</strong></h3><p>Before you test the load balancing, make sure both of your backend servers are working properly.</p><p><strong>- On Server 1</strong>, create a simple HTML file that says “Hello from Server 1”:</p><div><pre><code>sudo nano /usr/share/nginx/html/index.html
</code></pre></div><p><strong>- Write: Hello from Server 2 n the file.</strong></p><p><strong>- On Server 2, do the same but say “Hello from Server 2”.</strong></p><p><strong>- Test each server directly:</strong></p><p>From your load balancer instance, you can use curl to check if each server is serving the correct page:</p><div><pre><code>curl http://172.31.27.145  # Server 1
curl http://172.31.27.91   # Server 2
</code></pre></div><p>You should see “Hello from Server 1” for Server 1 and “Hello from Server 2” for Server 2.</p><h3><strong>Step 5: Test the Load Balancer</strong></h3><p>Now, let’s put the load balancing to the test. Open your browser and go to the <strong>public IP</strong> of the <strong>Load Balancer</strong>.</p><p>When you hit the load balancer’s IP:</p><ul><li>It should alternate between “Hello from Server 1” and “Hello from Server 2” as Nginx sends traffic to each backend server in turn.</li></ul><h1><strong>Conclusion: You Did It!</strong></h1><p>Congratulations! ? You’ve just set up a basic load balancing system using Nginx on AWS EC2 instances. By distributing traffic across multiple servers, you’ve made your web app faster, more reliable, and more scalable.</p><p>This is just the beginning — you can easily expand this setup by adding more backend servers or even implementing more advanced load balancing techniques.</p><h1><strong>What's Next?</strong></h1><p><strong>1. Add More Servers</strong>: Scale up by adding more backend servers to handle more traffic.</p><p><strong>2. Secure Your Setup</strong>: Set up SSL certificates to encrypt traffic.</p><p><strong>3. Monitor Your Servers</strong>: Use AWS CloudWatch or Nginx logs to monitor the performance of your servers and load balancer.</p><p>Now that you understand how load balancing works, you can implement it in your projects to keep your websites and applications fast and responsive, no matter how much traffic you get! ??</p>

You’ve built an amazing website or app, and the traffic starts pouring in. Users are clicking, scrolling, and browsing like never before. Everything is running smoothly… until it’s not. Suddenly, your server gets overwhelmed, and your site slows down. Worse, it crashes. Yikes!  What do you do?  Enter Load Balancing — your superhero in the world of web traffic. ?  But why do we need it, and how do we set it up on AWS using Nginx? Let's dive in and break it down in simple terms.

How to Set Up a Load Balancer in AWS: A Simple Guide to Keep Your Website Fast and Scalable

<h1><strong>What Is Rate Limiting?</strong></h1><p>At its core, <strong>rate limiting</strong> is a control mechanism used in software systems, especially APIs, to restrict how many requests a client can make within a specific timeframe.</p><p>Think of it as setting the speed limit on a highway. Without it, cars (or requests) might flood the lanes, causing congestion (or a system crash). Rate limiting ensures everyone gets to their destination (or data) without overwhelming the system.</p><h1><strong>Why Does Rate Limiting Matter?</strong></h1><p>Imagine running an online service where thousands (or even millions) of users access your API. What happens if one rogue user floods your system with excessive requests?</p><p>Without rate limiting, here’s what you might face:</p><p><strong>1.Server Overload:</strong> Your system might slow down or crash entirely.</p><p><strong>2.Unhappy Users:</strong> Other users won’t get timely responses, leading to frustration.</p><p><strong>3.Increased Costs:</strong> Handling unnecessary requests eats up resources.</p><p><strong>4.Security Risks:</strong> It’s an open invitation for <strong>DDoS (Distributed Denial of Service)</strong> attacks.</p><h1><strong>How Does Rate Limiting Work?</strong></h1><p><img src="https://firebasestorage.googleapis.com/v0/b/personal-blog-darmajr.appspot.com/o/blog-content%2Frate_limiter_flow.jpg?alt=media&amp;token=0150c280-c4a3-42b2-8422-0a223711a465" alt="Rate limieter workflow example" width="720px"></p><h2><strong>1. The Client Sends a Request</strong></h2><p>Let’s start with the clients—your users. They might be requesting to fetch data, submit a form, or interact with your app in some way. Every action sends a request to your API server.</p><p>Now, without a system in place, too many requests from too many clients could crush the API. This is where the rate limiter steps in.</p><h2><strong>2. The Rate Limiter Checks the Gate</strong></h2><p>The rate limiter is your vigilant bouncer. Each incoming request is checked against a set of rules. For example:</p><ul><li><strong>Rule:</strong> No more than 10 requests per second per client.</li><li><strong>Rule:</strong> A maximum of 1,000 requests per day for premium users.</li></ul><p>If a request fits within the rules, it gets a thumbs-up. If not, the rate limiter steps in with a polite "Sorry, you've reached your limit" (a.k.a., the <code>429 Too Many Requests</code> error).</p><h2><strong>3. Redis: The Silent Helper</strong></h2><p>Now, how does the rate limiter keep track of all this? Enter <strong>Redis</strong>, the speedy memory store.</p><p>Redis is like a super-efficient notebook that logs each client’s request count. Here’s how it works:</p><p>When a request comes in, Redis:</p><p>-Checks how many requests the client has already made.</p><p>-Updates the tally in real-time.</p><p>Redis’s speed and scalability make it perfect for handling this kind of workload.</p><h2><strong>4. Forwarding to the API Server</strong></h2><p>If the request passes the rate limiter’s scrutiny, it’s sent to the <strong>API server</strong>.</p><p>The server processes the request, performs the required action (like retrieving data or updating a record), and sends the response back to the client.</p><h1><strong>Common Rate Limiting Strategies</strong></h1><p>Here are some popular methods to implement rate limiting:</p><h3>1. <strong>Fixed Window Algorithm</strong></h3><p>Think of it as a time bucket. If you allow 100 requests per minute, the count resets every minute.</p><p><strong>Example:</strong></p><p>If a user sends 99 requests in the last second of a window and 100 in the next second, they technically make 199 requests within two seconds. (Uh-oh!)</p><h3>2. <strong>Sliding Window Algorithm</strong></h3><p>This method smooths things out by tracking requests over a rolling time window. It’s like always looking back 60 seconds from the current moment to count requests.</p><h3>3. <strong>Token Bucket</strong></h3><p>Imagine each user has a bucket filled with tokens. Each request consumes a token. If the bucket is empty, no more requests are processed until it refills.</p><h3>4. <strong>Leaky Bucket</strong></h3><p>This works like a dripping faucet. Even if the user sends requests in bursts, the system processes them at a consistent rate.</p><h1><strong>Where Is Rate Limiting Used?</strong></h1><p>Rate limiting isn’t just for APIs—it’s everywhere!</p><p><strong>1. Social Media Platforms:</strong> To prevent spamming or abuse (e.g., limiting tweets per minute).</p><p><strong>2. E-Commerce Sites:</strong> To stop bots from sniping deals during flash sales.</p><p><strong>3. Gaming Servers:</strong> To ensure fair play and prevent server overloads.</p><p><strong>4. Banking APIs:</strong> To protect sensitive systems from fraud or misuse.</p><h1><strong>Why Rate Limiting is Essential</strong></h1><p><strong>Rate limiting isn’t just about saying “no.” It’s about balance.</strong></p><p>Here’s what it brings to the table:</p><p><strong>1. Fair Access:</strong> Every client gets a fair chance to use the API without hogging resources.</p><p><strong>2. Protection:</strong> Prevents accidental overloads or deliberate attacks (like DDoS) from crashing the system.</p><p><strong>3. Cost Efficiency:</strong> By controlling traffic, you reduce server strain and save on infrastructure costs.</p><h1><strong>The Big Picture</strong></h1><p>With rate limiting, APIs can breathe easy, knowing that they’re protected from chaos while serving users efficiently. It’s not just a technical tool—it’s a safeguard for smooth operations.</p><p>So, next time you’re designing an API or interacting with one, remember: there’s a silent hero ensuring everything runs seamlessly. Whether it’s Redis handling the count or the rate limiter enforcing rules, this system is your API’s best friend.</p>

Imagine you’re hosting a party, and everyone wants to grab snacks from the buffet table at the same time. It’s chaos! Some guests get everything they want, while others leave empty-handed. What if you had a rule where each guest could only take two items at a time? Suddenly, everyone gets a fair share, and your party doesn’t turn into a food fight. That’s rate limiting in a nutshell!  But what exactly is rate limiting, and why is it so important? Let’s dive in and explore this concept together.

Understanding Rate Limiting: A Guide to Staying in Control of Your APIs

In this lab we will implement simple CQRS architecture pattern using apache kafka as a message broker, elastic search as a search service and mysql database as a command service.

Building a Simple CQRS Pattern Architecture

<p>Micro frontends are an architectural approach that applies the principles of microservices to frontend development. Instead of building a single, monolithic frontend application, the user interface is divided into smaller, independent pieces called <strong>micro frontends</strong>, each of which is developed, deployed, and maintained independently.</p><h1><strong>The Problem with Monolithic Frontends</strong></h1><p>A <strong>monolithic frontend</strong> refers to a single, large codebase that manages the entire user interface of an application. While this approach works well for small applications, it becomes increasingly challenging to manage and scale as the application and development teams grow. Below are the key problems associated with monolithic frontends:</p><h2><strong>1. Scaling Teams</strong></h2><ul><li><strong>Coordination Overhead</strong>: In a large team, multiple developers work on the same codebase. This can lead to frequent merge conflicts, delayed pull requests, and dependency issues.</li><li><strong>Limited Parallel Development</strong>: Because the codebase is tightly coupled, teams cannot work independently on different parts of the application without stepping on each other's toes.</li></ul><h2><strong>2. Slower Development Cycles</strong></h2><ul><li><strong>Single Deployment Pipeline</strong>: In a monolithic frontend, all changes must pass through the same build and deployment pipeline. This means Small changes (e.g., fixing a typo) require deploying the entire application, also A single bug can block the entire release process.</li><li><strong>Longer Testing Time</strong>: The larger the application, the more time and effort it takes to ensure that new changes don’t break existing functionality.</li></ul><h2><strong>3. High Risk of Changes</strong></h2><ul><li><strong>Ripple Effect</strong>: Since everything is interconnected, even small changes in one part of the application can have unintended consequences elsewhere.</li><li><strong>Rollback Challenges</strong>: If something goes wrong after deployment, rolling back requires reverting the entire application, not just the problematic component.</li></ul><h2><strong>4. Tech Stack Lock-In</strong></h2><ul><li><strong>Difficult to Adopt New Frameworks</strong>: In a monolithic frontend, the entire application is built using a single framework or library. Upgrading or switching technologies is a monumental task that may require rewriting the entire application. <strong>Example</strong>: Migrating from AngularJS to React would involve significant effort and downtime.</li><li><strong>No Flexibility for Teams</strong>: Teams must stick to the same tech stack, even if certain parts of the application would benefit from newer or more suitable tools.</li></ul><h1><strong>How Micro Frontends Work</strong></h1><p>Micro frontends operate on the principle of <strong>divide and conquer</strong>. Instead of managing one gigantic codebase, you divide your application into smaller units.</p><p>The idea behind micro frontends is simple yet powerful:</p><p><strong>1. Divide</strong> your application into smaller, self-contained pieces.</p><p><strong>2. Conquer</strong> by developing, testing, and deploying these pieces independently.</p><p>Instead of one massive codebase where every change has the potential to disrupt the entire application, you get a collection of smaller, focused units that can evolve at their own pace. Here is the breakdown of how divide and conquer works in microfrontend.</p><p><br></p><p><img src="https://firebasestorage.googleapis.com/v0/b/personal-blog-darmajr.appspot.com/o/blog-content%2Fmonolit_vs_microfrontend.png?alt=media&amp;token=24eba3d8-b7f3-4674-b33e-62daf0517afd" alt="Monolithic Vs Microfrontend" width="720px"></p><p><br></p><p>In a traditional monolithic architecture (left side), all components of a web application are tightly coupled and deployed as a single unit. This includes the web application layer (frontend), integration layer (APIs), and service layer (backend).</p><p>Microfrontends (right side) break down a large web application into smaller, independent frontend applications (also called "micro-applications" or "micro-frontends"). Each microfrontend is responsible for a specific feature or section of the application. This approach promotes modularity, scalability, and faster development cycles.</p><p>The divide and conquer approach is central to microfrontends. It involves:</p><h3><strong>Dividing the Application:</strong></h3><ol><li><strong>Identify Features:</strong> First, break down the application into distinct features or sections. For example, in the image, we have "Cart," "Website," and "Payment" as separate features.</li><li><strong>Assign Teams:</strong> Each feature is assigned to an independent team. This allows teams to work autonomously, focusing on their specific feature.</li></ol><h3><strong>Conquering the Features:</strong></h3><ol><li><strong>Independent Development:</strong> Each team develops its feature as a standalone frontend application. They can use different technologies (React, JAML, etc.) and frameworks as needed.</li><li><strong>API Integration:</strong> Teams define clear APIs for their microfrontends to communicate with each other and with the backend services. This ensures loose coupling and flexibility.</li></ol><h3><strong>Composition and Orchestration:</strong></h3><ol><li><strong>Frameworks and Tools:</strong> A framework or library is used to combine the microfrontends into a cohesive user experience. This can be done using techniques like server-side composition (e.g., with a Node.js server) or client-side composition (e.g., with a JavaScript framework like Single-SPA).</li><li><strong>Routing and Navigation:</strong> The framework handles routing and navigation between microfrontends, ensuring a seamless user experience.</li><li><strong>Shared Components:</strong> If necessary, shared components can be developed and used across multiple microfrontends. This promotes consistency and reduces code duplication.</li></ol><h1>Conclusion</h1><p>Micro frontends represent a modern approach to building scalable, resilient, and maintainable frontend applications. By breaking down your application into smaller, independent pieces, you empower your teams to innovate faster and reduce deployment risks.. If you’ve ever felt constrained by the limitations of a monolithic frontend, it might be time to explore the possibilities of micro frontends. They’re not just a technical solution—they’re a way to rethink how we build for the web.</p>

Have you ever worked on a massive frontend application where every change felt risky and deploying updates took ages? If so, micro frontends might be the solution you’ve been looking for

The concept of splitting a frontend into smaller, manageable pieces.

<div id="content-0"><p>In today's rapidly evolving technological landscape, applications are becoming increasingly complex and distributed. To manage this complexity and ensure high availability, reliability, and scalability, organizations are turning to Kubernetes. This powerful container orchestration platform has revolutionized the way we deploy and manage applications.</p></div><div id="content-1"><h1>Why Kubernetes?</h1></div><div id="content-2"><p><strong>1. Simplified Deployment and Management:</strong></p><ul><li><strong>Automated Deployment:</strong> Kubernetes automates the deployment process, eliminating manual intervention and reducing the risk of human error. With a few configuration changes, you can deploy complex applications to multiple environments with ease.</li><li><strong>Self-Healing:</strong> Kubernetes can automatically detect and recover from failures, ensuring that your applications remain up and running. If a pod fails, Kubernetes will automatically restart it on a different node.</li><li><strong>Scalability:</strong> You can easily scale your applications up or down to meet changing demand, without requiring significant manual effort. Whether it's a sudden traffic spike or a planned scaling event, Kubernetes can handle it seamlessly.</li></ul></div><div id="content-3"><p><strong>2. Efficient Resource Utilization:</strong></p><ul><li><strong>Resource Allocation:</strong> Kubernetes efficiently allocates resources (CPU, memory) to your applications, maximizing utilization and minimizing waste. It ensures that your applications get the resources they need, while avoiding overprovisioning.</li><li><strong>Dynamic Scheduling:</strong> It intelligently schedules pods onto nodes, optimizing resource allocation across the cluster. This ensures that your applications are always running on the most suitable nodes, regardless of their resource requirements.</li></ul><p><strong>3. Enhanced Reliability and Availability:</strong></p><ul><li><strong>High Availability:</strong> Kubernetes ensures high availability by replicating your applications across multiple nodes, providing redundancy and fault tolerance. If one node fails, your application will continue to run on other nodes.</li><li><strong>Load Balancing:</strong> It automatically distributes traffic across multiple instances of your application, improving performance and reliability. This ensures that no single instance is overwhelmed, and your users get a consistent experience.</li></ul><p><strong>4. Increased Flexibility and Portability:</strong></p><ul><li><strong>Container-Based:</strong> Kubernetes is container-based, allowing you to package your applications and their dependencies into portable units. This makes it easy to move your applications between different environments, such as development, testing, and production.</li><li><strong>Platform Agnostic:</strong> It can run on various infrastructure platforms, including public clouds (AWS, Azure, GCP), private clouds, and on-premises data centers. This gives you the flexibility to choose the best platform for your needs.</li></ul><p><br></p></div><div id="content-4"><h1>Kubernetes Architecture</h1></div><div id="content-6"><img style='width:720px;' src='https://firebasestorage.googleapis.com/v0/b/personal-blog-darmajr.appspot.com/o/blog-content%2Fkubernetes-architecture.png?alt=media&token=cd32c87e-d584-4aec-a83f-f0c20d7d0f5c'/></div><div id="content-7"><p>This diagram provides a clear overview of the key components in a Kubernetes cluster and how they interact with each other. Let's break down each component:</p><p><strong>Control Plane</strong></p><ul><li><strong>API Server:</strong> The main entry point for all interactions with the cluster. All requests (e.g., creating a new pod, scaling a deployment) are sent to the API Server.</li><li><strong>etcd:</strong> A distributed, consistent, highly-available key-value store that stores the entire cluster state. All information about pods, services, deployments, and more is stored here.</li><li><strong>Controller Manager:</strong> Manages various controllers responsible for ensuring the cluster is in the desired state. Examples of controllers include Deployment Controller, ReplicaSet Controller, and Job Controller.</li><li><strong>Scheduler:</strong> Responsible for scheduling pods to available nodes. It considers various factors like resource availability, affinities, and anti-affinities.</li></ul><p><strong>Node</strong></p><ul><li><strong>Kubelet:</strong> An agent that runs on each node. Kubelet ensures that the containers specified in pod manifests are running on the node.</li><li><strong>kube-proxy:</strong> A network proxy that implements network rules for services and load balancing.</li></ul><p><strong>Component Interactions</strong></p><ol><li><strong>User or Tool:</strong> When you want to create or manage Kubernetes resources (e.g., using kubectl), you interact with the API Server.</li><li><strong>API Server:</strong> Receives the request, validates it, and stores it in etcd.</li><li><strong>Controller Manager:</strong> Monitors changes in etcd and takes necessary actions. For example, if the number of replicas for a deployment doesn't match the desired state, the controller will create or delete pods.</li><li><strong>Scheduler:</strong> When there's a new pod to be scheduled, the scheduler selects the most suitable node and informs the Kubelet.</li><li><strong>Kubelet:</strong> Receives information from the scheduler and starts running the pod's containers.</li><li><strong>kube-proxy:</strong> Manages networking to ensure traffic is routed to the correct pods.</li></ol><p><br></p></div><div id="content-8"><h1><strong>Deployments: Scale, Update, Rollback</strong></h1><p>Imagine you have a simple web application (e.g., a Node.js app) running in a Kubernetes cluster. The application is exposed via a Kubernetes Service, and you want to manage it using a Deployment. Here’s how you can implement this:</p></div><div id="content-9"><h2>1. <strong>Creating a Deployment</strong></h2><p>First, you'll create a Deployment to manage your application.</p></div><div id="content-10"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-web-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-web-app
  template:
    metadata:
      labels:
        app: my-web-app
    spec:
      containers:
        - name: my-web-app
          image: myusername/my-web-app:1.0
          ports:
            - containerPort: 80</code></pre></div><div id="content-11"><ul><li><strong>Replicas</strong>: This specifies how many pods you want to run.</li><li><strong>Selector</strong>: This defines how to identify the pods managed by this Deployment.</li><li><strong>Template</strong>: This describes the pods that will be created.</li></ul><p><strong>Deploying the Application</strong>: Apply the Deployment with the following command:</p></div><div id="content-12"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">kubectl apply -f deployment.yaml</code></pre></div><div id="content-13"><h2>2. <strong>Scaling the Application</strong></h2><p>If you want to handle increased traffic, you can scale your Deployment up or down.</p><p><strong>Scaling Up</strong>: To increase the number of replicas to 5:</p></div><div id="content-16"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">kubectl scale deployment my-web-app --replicas=5</code></pre></div><div id="content-17"><p>Scaling Down: To decrease the number of replicas back to 3:</p></div><div id="content-18"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">kubectl scale deployment my-web-app --replicas=3</code></pre></div><div id="content-19"><h2>3. <strong>Updating the Application</strong></h2><p>When you want to update your application (for example, deploying a new version of the image), modify the Deployment:</p></div><div id="content-21"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">spec:
  template:
    spec:
      containers:
        - name: my-web-app
          image: myusername/my-web-app:2.0 # Updated version</code></pre></div><div id="content-22"><p>Applying the Update: You can update the Deployment by reapplying the configuration:</p></div><div id="content-23"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">kubectl apply -f deployment.yaml</code></pre></div><div id="content-24"><p>Kubernetes will perform a rolling update, gradually replacing the old pods with new ones.</p><h2>4. <strong>Checking the Update Status</strong></h2><p>To monitor the status of the update, use:</p></div><div id="content-25"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">kubectl rollout status deployment/my-web-app</code></pre></div><div id="content-26"><h2>5. <strong>Rolling Back an Update</strong></h2><p>If something goes wrong with the new version, you can roll back to the previous version easily:</p></div><div id="content-27"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">kubectl rollout undo deployment/my-web-app</code></pre></div><div id="content-28"><p>To check the history of the revisions, you can use:</p></div><div id="content-29"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">kubectl rollout history deployment/my-web-app</code></pre></div><div id="content-30"><h2>6. <strong>Verifying the Rollback</strong></h2><p>After rolling back, you can verify that the previous version is running:</p></div><div id="content-31"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">kubectl get deployments
kubectl describe deployment my-web-app</code></pre></div><div id="content-32"><h2>7. <strong>Creating a Service</strong></h2><p>A Kubernetes <strong>Service</strong> is used to expose your application, making it accessible from outside the cluster (or within, depending on your requirements). Here's an example of a Service configuration:</p></div><div id="content-33"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">apiVersion: v1
kind: Service
metadata:
  name: my-web-app-service
spec:
  selector:
    app: my-web-app
  ports:
    - protocol: TCP
      port: 80       # Port on the Service
      targetPort: 80 # Port on the container
  type: LoadBalancer</code></pre></div><div id="content-34"><h1><strong>In Conclusion:</strong></h1><p>Kubernetes architecture is designed to simplify the management of large-scale containerized applications. By understanding its components and interactions, you can effectively leverage Kubernetes to build reliable and scalable applications.</p></div>

In today's fast-paced digital landscape, applications are becoming increasingly complex and distributed. To manage this complexity and ensure high availability, reliability, and scalability, organizations are turning to Kubernetes. This powerful container orchestration platform has revolutionized the way we deploy and manage applications.

Why You Need Kubernetes: A Comprehensive Guide

<div id="content-0"><h1>Spark Architecture</h1><p>Imagine a Spark application as a bustling city. At the heart of this city is the <strong>Driver Program</strong>, which acts like the mayor overseeing everything that happens. The driver program is responsible for running your code, coordinating work, and making key decisions about how tasks should be executed. Like a city planner, it organizes and manages the tasks to be performed by the cluster, breaking down large jobs into smaller units of work. These jobs are then divided into <strong>tasks</strong> that Spark can run in parallel. The beauty of this architecture lies in its ability to scale, allowing multiple tasks to be completed simultaneously on different data partitions. These tasks are dispatched to <strong>executors</strong>, which are the hard workers of the cluster, doing the heavy lifting.</p></div><div id="content-4"><img style='width:720px;' src='https://firebasestorage.googleapis.com/v0/b/personal-blog-darmajr.appspot.com/o/blog-content%2FApache-Spark-architecture.png?alt=media&token=ae8b882a-a359-4e63-aad6-a9a9f0a06ff8'/></div><div id="content-5"><p>Executors are independent workers spread across the cluster, each taking responsibility for a portion of the work. They are not just performing tasks but are also responsible for caching data, which can speed up future computations by reusing cached data rather than starting from scratch. Much like how a construction team works efficiently by reusing tools and materials at the job site, executors are optimized to perform computations without redundant effort.</p><p>To help you better visualize this, think of a project that requires breaking down and shipping parts to different locations. Each part can be processed independently and reassembled once all pieces are completed. The driver program ensures these tasks are coordinated correctly, and executors handle each piece of the job.</p><p>But what’s the relationship between <strong>jobs</strong>, <strong>tasks</strong>, and <strong>stages</strong>? Imagine you're the head of a large construction project, and you need to break down the overall project (the <strong>job</strong>) into manageable tasks for each construction team. A <strong>task</strong> in Spark operates on a specific partition of data, much like a construction team focusing on a particular section of the building. Once a group of tasks that don’t depend on any other data is identified, Spark bundles them into <strong>stages</strong>. A stage represents a set of tasks that can be executed independently without needing data from elsewhere in the project. However, sometimes a task will need information from another part of the dataset, requiring what’s known as a <strong>data shuffle</strong>. This shuffle is like coordinating deliveries between different construction teams—an operation that can slow things down due to the necessary data exchange, but essential for the overall completion of the job.</p><p>What happens when Spark needs to run in different environments? That’s where <strong>cluster modes</strong> come in. There are several ways Spark can be deployed, each suited to different use cases. The simplest is <strong>local mode</strong>, ideal for testing on your own machine. Imagine local mode as running your city’s planning department with just one employee—the driver program manages everything on its own, without help from external workers. This is great for testing, but when you need real performance, it’s time to deploy Spark in a cluster.</p><p>In a full cluster setup, Spark supports several modes. <strong>Spark Standalone</strong> is the quickest way to set up a cluster environment, ideal for smaller projects or when you want complete control over the infrastructure. For those already working in large-scale environments with Hadoop, <strong>YARN</strong> is a natural choice. It integrates seamlessly with the broader Hadoop ecosystem, making it easy to manage resources. For greater flexibility and the ability to handle dynamic workloads, <strong>Apache Mesos</strong> comes into play, providing a more robust partitioning and scaling system.</p><p>But if you’re looking for a modern, cloud-native approach, <strong>Kubernetes</strong> offers powerful benefits. Running Spark on Kubernetes is like managing a city that can grow or shrink as needed, using containers to deploy and scale your Spark applications. With Kubernetes, Spark becomes highly portable, making it easy to run your Spark jobs in any cloud environment. You can set up your Spark application inside containers and have them scale automatically based on demand, ensuring smooth processing even as workloads increase.</p></div><div id="content-6"><p><br></p><h1>Mastering the Art of Running Apache Spark Applications</h1><p>Ever wondered how to launch your Apache Spark application into action? Whether you’re processing mountains of data or just running local tests, Apache Spark’s flexibility makes it incredibly powerful. But let’s be real—understanding how to run a Spark application might seem daunting at first. Fear not! Here’s your step-by-step guide to Spark mastery.</p><p>At the heart of it all is the <code>spark-submit</code> script. Think of it as Spark’s personal conductor, ensuring your application runs smoothly across a distributed cluster or right on your local machine. With <code>spark-submit</code>, you’ve got full control: it lets you specify everything from the cluster manager you want to connect to, to how much memory and CPU cores your application needs. You can also include any additional files or libraries your app requires, making sure all the pieces are in place for a flawless run.</p><p>Now, let’s talk dependencies. In Spark, making sure the driver and executors have access to the right libraries is crucial. If you’re using Java or Scala, bundling all your code and libraries into a single uber-JAR (or fat JAR) is a common approach. This neat package ensures that everything is shipped out and accessible where it’s needed. For Python applications—aka PySpark—you’ll want to ensure that each node in your cluster has the exact same Python libraries installed. Imagine trying to run a marathon with mismatched shoes—it won’t end well, right? Same idea with your Spark dependencies.</p><p>If you’re in the mood for a more hands-on, experimental approach, then the <strong>Spark Shell</strong> is your playground. This interactive tool lets you dive right into Spark with either Scala or Python, without needing to write and submit an entire application. When you fire up the Spark Shell, it automatically sets up everything for you—giving you instant access to Spark’s APIs. You can run quick computations, play around with datasets, and see the results in real-time, making it perfect for debugging or just satisfying your curiosity.</p><p>So, to sum it all up: running an Apache Spark application is as easy as using <code>spark-submit</code> to launch your code, bundling your dependencies into an uber-JAR (or ensuring Python libraries are ready), and—if you're feeling adventurous—jumping into the Spark Shell for some interactive magic. Spark truly puts the power of distributed computing at your fingertips.</p></div><div id="content-7"><p>We now trying to submit Apache Spark applications from a python script. This exercise is straightforward thanks to Docker Compose. In this lab, you will:</p><ul><li>Install a Spark Master and Worker using Docker Compose</li><li>Create a python script containing a spark job</li><li>Submit the job to the cluster directly from python (Note: you’ll learn how to submit a job from the command line in the Kubernetes Lab)</li></ul><p><br></p></div><div id="content-8"><h2>Install a Apache Spark cluster using Docker Compose</h2></div><div id="content-9"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">git clone https://github.com/big-data-europe/docker-spark</code></pre></div><div id="content-11"><p>change to that directory and attempt to docker-compose up</p></div><div id="content-12"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">cd docker-spark
docker-compose up</code></pre></div><div id="content-13"><p>After quite some time you should see the following message:</p><p><em>Successfully registered with master spark://&lt;server address&gt;:7077</em></p></div><div id="content-14"><h2>Create Code</h2></div><div id="content-15"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">import findspark
findspark.init()
from pyspark import SparkContext, SparkConf
from pyspark.sql import SparkSession
from pyspark.sql.types import StructField, StructType, IntegerType, StringType
sc = SparkContext.getOrCreate(SparkConf().setMaster('spark://localhost:7077'))
sc.setLogLevel("INFO")
spark = SparkSession.builder.getOrCreate()
spark = SparkSession.builder.getOrCreate()
df = spark.createDataFrame(
    [
        (1, "foo"),
        (2, "bar"),
    ],
    StructType(
        [
            StructField("id", IntegerType(), False),
            StructField("txt", StringType(), False),
        ]
    ),
)
print(df.dtypes)
df.show()</code></pre></div><div id="content-16"><h2><strong>Execute code / submit Spark job</strong></h2><p>Now we execute the python file we saved earlier.</p><p>In the terminal, run the following commands to upgrade the pip installer to ensure you have the latest version by running the following commands.Now we execute the python file we saved earlier.</p></div><div id="content-17"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">rm -r ~/.cache/pip/selfcheck/
pip3 install --upgrade pip
pip install --upgrade distro-info</code></pre></div><div id="content-18"><p>Please enter the following commands in the terminal to download the spark environment.</p></div><div id="content-20"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">wget https://archive.apache.org/dist/spark/spark-3.3.3/spark-3.3.3-bin-hadoop3.tgz 
&& tar xf spark-3.3.3-bin-hadoop3.tgz && rm -rf spark-3.3.3-bin-hadoop3.tgz</code></pre></div><div id="content-21"><p>Run the following commands to set up the&nbsp;&nbsp;which is preinstalled in the environment and&nbsp;&nbsp;which you just downloaded.</p></div><div id="content-22"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">export JAVA_HOME=/usr/lib/jvm/java-1.11.0-openjdk-amd64
export SPARK_HOME=/home/project/spark-3.3.3-bin-hadoop3</code></pre></div><div id="content-23"><p>Install the required packages to set up the spark environment.</p></div><div id="content-24"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">pip install pyspark
python3 -m pip install findspark</code></pre></div><div id="content-25"><p>Type in the following command in the terminal to execute the Python script.</p></div><div id="content-26"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">python3 submit.py</code></pre></div><div id="content-27"><p>go to port 8080 to see the admin UI of the Spark master</p></div><div id="content-28"><img style='width:720px;' src='https://firebasestorage.googleapis.com/v0/b/personal-blog-darmajr.appspot.com/o/blog-content%2Fdemo-spark-submit.png?alt=media&token=b8fe9a3e-6bcf-42f4-82a7-2b1842b11e74'/></div><div id="content-29"><p><strong>Look at that!</strong> You can now see all your registered workers (we’ve got one for now) and the jobs you’ve submitted (just one at the moment) through Spark's slick interface. Want to dig even deeper? You can access the worker’s UI by heading to port 8081 and see what’s happening under the hood!</p><p>In this hands-on lab, you've set up your very own experimental Apache Spark cluster using Docker Compose. How cool is that? Now, you can easily submit Spark jobs directly from your Python code like a pro!&nbsp;</p><p>But wait, there’s more! In our next adventure—the Kubernetes lab—you’ll unlock the power of submitting Spark jobs right from the command line.</p></div>

Welcome to an in-depth exploration of Apache Spark’s architecture! Whether you’re new to Spark or looking to refresh your understanding, this interactive guide will walk you through the key concepts that power Spark’s ability to process massive datasets quickly and efficiently.

Mastering Apache Spark: An Engaging Dive into Its Architecture and Clusters

<div id="content-0"><h1>MapReduce: Powering Big Data Processing in Hadoop</h1><p>In the world of Big Data, <strong>MapReduce</strong> stands as one of the key mechanisms for efficiently processing enormous datasets across distributed systems. It enables parallel processing of data across multiple computers within a cluster, making data handling at scale fast and reliable.</p><p>Let’s make it easier to grasp and dive into how MapReduce works in a more engaging, reader-friendly way.</p></div><div id="content-1"><h2>What Exactly is MapReduce?</h2><p>Imagine you have a huge task that needs to be done, but you can split it up among several friends. Each of them works on a piece of the task, and when they’re done, someone else comes along to gather all the pieces, combine them, and finish the job. That’s essentially how MapReduce works in Hadoop.</p><p>In simple terms:</p><ol><li><strong>Map</strong>: The data gets divided into smaller, manageable pieces, and each piece is processed independently. The goal here is to transform data into key-value pairs.</li><li><strong>Reduce</strong>: After the data is organized, the pieces with the same key are grouped together. Then, the reduce function steps in to summarize or aggregate the data to produce the final output.</li></ol><p>This parallel processing is what makes MapReduce so powerful in handling large datasets efficiently</p></div><div id="content-2"><h2>Setting the Right Number of Mappers and Reducers</h2><p>While Hadoop does a lot of the work for you, it’s useful to know how mappers and reducers are assigned, because it can directly affect performance.</p><h3><strong>How Many Mappers?</strong></h3><ul><li><strong>Automatically</strong>: Hadoop typically assigns one mapper per block of data in HDFS (Hadoop Distributed File System). Each block is handled by one mapper.</li><li><strong>Manually</strong>: If needed, you can adjust this manually by configuring the <code>mapreduce.job.maps</code> parameter in Hadoop settings. For example, if you want smaller or larger chunks of data per mapper, this is where you can tweak it.</li></ul><h3><strong>How Many Reducers?</strong></h3><ul><li><strong>Manual Configuration</strong>: Reducers are usually set by you, the developer, when writing the MapReduce job. You specify how many reducers are needed with the <code>mapreduce.job.reduces</code> setting.</li><li><strong>Dynamic Adjustment</strong>: Sometimes, Hadoop can adjust the number of reducers based on the data size, but setting it manually is common for optimizing performance.</li></ul><p>More mappers or reducers can mean more parallel processing, but it’s important to find the right balance to avoid bottlenecks during the shuffle and sort phase (when data is grouped before reduction).</p></div><div id="content-4"><h2>The MapReduce Workflow in Action</h2><p>Let’s look at a real-life example of how MapReduce works, step-by-step, using Hadoop. Let’s say you want to process a file called <code>jar.txt</code>. Here’s how it works:</p><h3>1. <strong>Input Data to HDFS</strong></h3><p>You (the client) upload the file <code>jar.txt</code> to Hadoop’s distributed file system (HDFS). Simple enough, but this is where the magic starts.</p><h3>2. <strong>JobTracker &amp; NameNode – The Commanders</strong></h3><p>The <strong>JobTracker</strong> (which coordinates all MapReduce jobs) reaches out to the <strong>NameNode</strong> (the master of HDFS) to find out where the file is stored. NameNode provides the metadata, explaining how the file will be divided into blocks and stored across DataNodes.</p><h3>3. <strong>Data Replication for Safety</strong></h3><p>For reliability, Hadoop replicates each block of data across multiple DataNodes. So, your file doesn’t just exist in one place—it’s copied to ensure that, even if one node goes down, your data is still accessible.</p><h3>4. <strong>TaskTracker &amp; Block Assignment</strong></h3><p>Each block of data is assigned to a <strong>TaskTracker</strong> running on a DataNode. These TaskTrackers are responsible for managing the mappers that will process the blocks. It’s like sending out your team of workers to handle different pieces of the puzzle.</p><h3>5. <strong>Map Task – Processing Begins</strong></h3><p>The TaskTrackers execute the map phase by processing the data blocks in parallel. Each mapper works on its own piece of data, transforming it into key-value pairs.</p><h3>6. <strong>Intermediate Results Stored Locally</strong></h3><p>Once the map phase is complete, the intermediate results are saved locally on each DataNode. These results aren’t final yet—they still need to be combined.</p><h3>7. <strong>Reduce Task – Bringing It All Together</strong></h3><p>Now, the <strong>Reduce</strong> phase begins. TaskTrackers running the reduce task gather the intermediate results from all mappers. They group the data by key and aggregate it to produce the final output.</p><h3>8. <strong>Final Output Stored in HDFS</strong></h3><p>Once the reducers finish their job, the final output is written back to HDFS. This means your processed data is now available in the distributed file system, ready for use.</p><h3>9. <strong>Job Completion – Success!</strong></h3><p>Finally, the JobTracker informs you that the job is complete, and you can access the results in HDFS. Mission accomplished!</p></div><div id="content-5"><h2>Why Should You Care About MapReduce?</h2><p>MapReduce simplifies the complex task of processing huge datasets by breaking it into smaller chunks and handling everything in parallel. It’s incredibly efficient and, thanks to data replication, fault-tolerant. Even if a node fails, your job won’t crash—the data is safe, and the job continues on other nodes.</p><p>So, the next time you’re dealing with a massive dataset, just remember—MapReduce is like having an army of helpers, all working together to get the job done fast!</p></div>

Ever wondered how companies handle mountains of data efficiently? Enter MapReduce—Hadoop’s superhero when it comes to processing large datasets. Instead of one machine trying to handle everything, MapReduce breaks the work into smaller chunks and distributes it across many machines, making the process faster and more reliable.

MapReduce: The Magic Behind Processing Big Data in Hadoop

<div id="content-0"><h1>How HDFS Stands Out from Traditional File Systems</h1><p>Have you ever wondered how Hadoop handles those massive datasets? It’s not quite like how your typical computer saves files using NTFS or FAT32. Let's break it down!</p></div><div id="content-1"><h1>What is HDFS?</h1><p>HDFS, or Hadoop Distributed File System, is the backbone of Hadoop. It’s specially built to handle huge volumes of data by spreading it across multiple machines, making it perfect for big data tasks. If you’ve got terabytes or even petabytes of information, HDFS can manage it efficiently.</p><p>Unlike regular file systems like <strong>NTFS</strong> (which you might find on your Windows laptop), HDFS isn't just about saving files locally. Instead, it breaks your files into chunks (called blocks), and then spreads them out over several machines. This allows Hadoop to process multiple pieces of data at the same time. Pretty neat, right?</p></div><div id="content-2"><img style='width:720px;' src='https://firebasestorage.googleapis.com/v0/b/personal-blog-darmajr.appspot.com/o/blog-content%2Fblock_and_replication.jpg?alt=media&token=7c465fa0-f2ed-444d-814a-1ab9c3e7c944'/></div><div id="content-3"><h1>The Magic of HDFS: Why Blocks and Replication Matter</h1><p>Here’s where HDFS gets really clever: it doesn’t just save your data in one place. It cuts files into large blocks, then duplicates (or replicates) those blocks across several machines. Why? So that if one machine crashes, your data isn’t lost. It’s all about fault tolerance.</p><p>Think of it like this: Imagine you’ve got a giant puzzle, but instead of keeping all the pieces in one box, you spread them across three different boxes. That way, even if you lose one box, you’ve still got enough pieces in the other two to complete the picture. That’s how replication in HDFS works. Each block is typically copied three times (though you can change that number if you need more or fewer backups).</p></div><div id="content-4"><h1>Comparing HDFS to NTFS</h1><p>Now, let’s compare this to NTFS. NTFS is a file system that works great for smaller files and doesn’t need to worry about distributing data over a network. When you save a file on your laptop, NTFS does its job locally—no fancy distribution happening here. If your hard drive fails, well, you better hope you’ve got a backup.</p><p>On the other hand, HDFS shines in situations where you're dealing with Big Data. Imagine trying to process a massive log file for an entire year across multiple servers. That’s when HDFS steps in to spread the workload across dozens or even hundreds of machines, making everything faster and more reliable.</p></div><div id="content-5"><h2>How Clusters Work in HDFS</h2><p>HDFS operates on a <strong>master-slave architecture</strong>, which, despite the name, is simpler than it sounds. Picture a group project where one person (the master) assigns tasks, and the rest (the slaves) carry them out. In Hadoop’s case, the <strong>NameNode</strong> is the master that keeps track of which machines (or <strong>DataNodes</strong>) are storing which blocks of data.</p><p>But here's the cool part: even if one of the DataNodes goes offline, you won't lose your data. Thanks to replication, other copies of the data are safe on different nodes. The NameNode makes sure that if something goes wrong, the system automatically reassigns tasks to healthy machines. So, your project continues without missing a beat.</p></div><div id="content-6"><h2>Blocks and Replication: The Heart of HDFS</h2><p>When you upload a file to HDFS, it doesn’t store the file in one piece. Instead, it breaks it into blocks—big ones, usually around 128 MB or 256 MB each (way bigger than what you'd see in NTFS or FAT32). These blocks are then distributed across multiple machines, and each block is typically replicated three times, just in case something goes wrong.</p><p>Let’s say you’ve uploaded a 1 GB file. HDFS would split that file into eight blocks, each around 128 MB. Then, it spreads these blocks out, saving them on different machines. And, for good measure, it replicates each block three times, storing those copies on different nodes. So even if one machine dies, the system still has two copies of every block.</p><p>This is what makes HDFS so powerful—by distributing both the data and the risk of failure, it allows Hadoop to manage enormous datasets without constantly worrying about losing data due to hardware issues.</p></div><div id="content-7"><h2>What Happens When You Write Data to HDFS?</h2><p>Here’s how the process goes when you write data to HDFS:</p><ol><li><strong>Request</strong>: You, or an application, send a request to HDFS to save some data.</li><li><strong>Communication</strong>: HDFS, through the <strong>NameNode</strong>, checks where your data should be stored. It’s like asking the master project manager where to put each piece of the puzzle.</li><li><strong>Breaking into Blocks</strong>: The data is then broken down into blocks and sent to different DataNodes, which store those blocks.</li><li><strong>Replication</strong>: As each block gets stored, it's also replicated across other nodes. If one copy goes missing, the system can recreate it from another.</li><li><strong>Confirmation</strong>: Once all blocks are safely stored and replicated, HDFS sends a confirmation, saying, “All good! Your data’s safely stored.”</li></ol><p>This process happens behind the scenes, but it's crucial to keeping your data safe and your big data applications running smoothly.</p></div><div id="content-9"><h2>What Happens When a DataNode Fails in HDFS?</h2><p>Imagine you're managing a vast library, and each book is split into multiple copies stored across different branches to ensure none of the information is lost. Now, what if one of the branches burns down? You wouldn't panic, right? Why? Because the other branches still hold copies. This is pretty much how Hadoop handles DataNode failures.</p><p>DataNodes store chunks of data, and it's inevitable that sometimes, things will go wrong. Maybe a server breaks down, or there's a network issue. But here’s the cool part: Hadoop has it all covered.</p></div><div id="content-10"><h3>How Does Hadoop Know a DataNode Failed?</h3><p>Think of it like a regular check-in system. Every few seconds, each DataNode sends a simple signal to the <strong>NameNode</strong> (the master controller) saying, “Hey, I’m here, and I’m working!” These signals are called <strong>heartbeats</strong>.</p><p>But what if the NameNode doesn’t hear from a DataNode? If it misses several heartbeats in a row, the NameNode starts to get suspicious. After a certain time, it officially declares, “This DataNode is down!”</p></div><div id="content-12"><h3>What Happens Next?</h3><p>Now, this is where Hadoop shows off its resilience. The moment a DataNode fails, the NameNode updates its metadata. Basically, it marks all the data blocks that were on that failed node as unavailable.</p><p>But remember—Hadoop doesn’t store just one copy of your data. It replicates blocks across multiple DataNodes. So, even though one DataNode has failed, the same data is safely stored on other DataNodes. The NameNode immediately gets to work, ensuring that all data remains fully backed up. It checks if any data blocks have dropped below the desired number of replicas (usually 3), and if so, it orders other healthy DataNodes to make fresh copies.</p></div><div id="content-13"><h3>Bringing a DataNode Back to Life</h3><p>Let’s say the failed DataNode eventually comes back online. It’s not like Hadoop just forgets about it. First, the NameNode checks whether the data stored there is still relevant. If the system has already made new replicas of those blocks, it might decide to remove the old, now-duplicated data to free up space.</p><p>Once the DataNode starts sending its heartbeats again, it gets reintegrated into the system, and life goes on as if nothing happened. It’s like having a team member who took a break and is now ready to get back to work!</p></div><div id="content-14"><p>In a nutshell, when a DataNode fails, Hadoop's architecture ensures there's no reason to worry. The NameNode quickly detects the failure, handles it behind the scenes by replicating data to other nodes, and everything continues running smoothly without missing a beat! Pretty smart, right?</p></div><div id="content-15"><h2>Wrapping Up</h2><p>HDFS might seem complex at first, but once you break it down, it’s really all about efficiency and reliability. By splitting files into blocks, distributing them across multiple machines, and replicating them for safety, HDFS ensures that your data is always accessible—even when things go wrong. So, if you're working with big datasets, HDFS is the perfect solution to keep everything running like clockwork.</p></div>

HDFS, or Hadoop Distributed File System, is the backbone of Hadoop. It’s specially built to handle huge volumes of data by spreading it across multiple machines, making it perfect for big data tasks.

Understanding Hadoop Distributed File System (HDFS)

<div id="content-0"><h1>What is Electron.js?</h1><p>Electron.js is an open-source framework that allows developers to build cross-platform desktop applications (Windows, macOS, and Linux) using web technologies like HTML, CSS, and JavaScript. Electron combines Chromium and Node.js so that developers can build desktop applications using the same technology they use for web development. I will discussed about the Modash project that i created too. So other people can use this template and ready to be launch as a producition app. This Modash project using the weird stack such as electron.js, React, Vite and IndexDb. i will tell you why i use this stack.</p><p><br></p><h1>Why Use React and Vite?</h1><p>React is a very popular JavaScript library for building user interfaces (UIs). By using React, we can create dynamic and interactive UI components and Vite is a build tool that provides speed and efficiency in frontend project development. Vite offers a fast development experience with hot module replacement (HMR) and a very fast build process.</p><p><br></p><h1>Why IndexedDB?</h1><p>IndexedDB is a web-based database storage solution that provides larger storage compared to LocalStorage and SessionStorage. IndexedDB is ideal for applications that require large data storage on the client side, such as desktop applications built with Electron.js. we can use other database instead of this indexDb in my template. but for a default i used this indexDB to manage my account login. this is not  a good practice because indexDb can be access trough the Web interface and directly clear the dabatase data. but this can be prevent in prouction by blocking the user to inspect the devtools. Here is the code to prevent user inspect and ruined their production app:</p><p><br></p><p><br></p></div><div id="content-1"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;"> if (process.env.NODE_ENV === "development") {
    win.loadURL("http://localhost:6969/");
    win.webContents.openDevTools();
  } else {
    win.loadFile(path.join(__dirname, "./dist/index.html"));

    // Remove the menu (including DevTools shortcuts)
    win.setMenu(null);

    // Disable context menu (right-click) that might have "Inspect Element"
    win.webContents.on('context-menu', (e) =&gt; {
      e.preventDefault();
    });

    // Optionally, prevent any manual attempts to open DevTools
    win.webContents.on('before-input-event', (event, input) =&gt; {
      if (input.key === 'I' && input.control && input.shift) {
        event.preventDefault();
      }
    });

    // Make sure DevTools are closed
    win.webContents.on('did-finish-load', () =&gt; {
      win.webContents.closeDevTools();
    });
  }
};</code></pre></div><div id="content-3"><p>This code snippet is part of an Electron application setup and handles different behaviors based on whether the application is running in development or production mode. In development mode, it loads the application from a local server (<code>http://localhost:6969/</code>) and opens the Developer Tools for debugging purposes. On the other hand, in production mode, it loads the application from a local HTML file, disables the menu, prevents the context menu from appearing (which might include the "Inspect Element" option), and disables the shortcut (<code>Ctrl+Shift+I</code>) for opening the Developer Tools. Additionally, it ensures that the Developer Tools are closed after the application finishes loading, contributing to a cleaner user interface and enhanced security.</p></div><div id="content-4"><h1>Modash</h1><p>This template is used to build the Dashboard application such as inventory list. sales, etc. the default template is using the React router dom for handling and manage the page. the starter kit for css framework i used is a Chakra-UI. You can visit this site to read Chakra-UI docs. <a href="https://v2.chakra-ui.com/" target="_blank">https://v2.chakra-ui.com/</a>. in this template i already built Home page. and the settings page. You can tweak and modify freely as you need if you want to build the dashboard app using this template. Here is the link of my github repository <a href="https://github.com/Barbarpotato/Modash" target="_blank">https://github.com/Barbarpotato/Modash</a>.&nbsp;</p></div><div id="content-5"><h1>Electron Problem with Production</h1><p>Sometimes Electron.js has some trouble to delivered as a production app if you used the electron.js and integrating it wirh React environment. i have face some problems in my journey such as error when production if i sed react-router-dom, etc. to prevent react-router-dom crash during a production, you can use the &lt;HashRouter&gt; instead of &lt;Browser Router&gt;. this will solve the issue in electron production app. Here is the example code:</p></div><div id="content-6"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">import React from 'react'
import ReactDOM from 'react-dom/client'
import { HashRouter } from 'react-router-dom'
import { ChakraProvider } from '@chakra-ui/react'
import theme from './theme/theme.js'
import App from './App.jsx'
import './index.css'
import { AuthProvider } from './hooks/useAuth.jsx'

ReactDOM.createRoot(document.getElementById('root')).render(
  &lt;ChakraProvider theme={theme}&gt;
    &lt;HashRouter&gt;
      &lt;AuthProvider&gt;
        &lt;App /&gt;
      &lt;/AuthProvider&gt;
    &lt;/HashRouter&gt;
  &lt;/ChakraProvider&gt;
)</code></pre></div><div id="content-7"><p><code>HashRouter</code> is a type of router provided by React Router DOM, a popular routing library for React applications. The <code>HashRouter</code> uses the hash portion of the URL (the part after the <code>#</code> symbol) to keep the UI in sync with the URL. This makes it suitable for applications that need to be deployed to servers that don't handle dynamic requests, such as GitHub Pages.</p></div><div id="content-8"><h1>Production Mode</h1><p>My Template is very ready about hearing the produciton mode. Dont worry about that. if you not optimis about my template during in production environment. You can clone my repository and attempt to make it as a production app. Go to a terminal and type <strong>npm run prod. </strong>This will make 2 folders name dist and out folder. where the dist folder is the build package for the react and the out folder is the application that you are going to install in your device. Here is the package.json file you can look and learn about the template environment:</p></div><div id="content-9"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">{
  "name": "modash-desktop-app",
  "productName": "Modash",
  "description": "Modification Dashboard Dekstop App for inventory management purpose",
  "private": true,
  "version": "1.0.0",
  "type": "module",
  "main": "electron.cjs",
  "scripts": {
    "dev": "cross-env NODE_ENV=development concurrently \"npm:serve\" \"npm:electron\"",
    "prod": "npm run build && npm run electron-build",
    "serve": "vite",
    "build": "cross-env NODE_ENV=production vite build",
    "preview": "vite preview",
    "electron": "wait-on tcp:6969 && electron .",
    "electron-build": "cross-env NODE_ENV=production electron-builder"
  },
  "build": {
    "appId": "electron-react-vite",
    "mac": {
      "icon": "public/download.ico"
    },
    "win": {
      "target": [
        "nsis"
      ],
      "icon": "public/download.ico"
    },
    "nsis": {
      "oneClick": false,
      "allowToChangeInstallationDirectory": true,
      "installerIcon": "public/download.ico",
      "uninstallerIcon": "public/download.ico",
      "uninstallDisplayName": "electron-react-vite"
    },
    "directories": {
      "output": "out"
    },
    "files": [
      "dist/**/*",
      "electron.cjs",
      "electron/**"
    ]
  },
  "dependencies": {
    "@chakra-ui/react": "^2.8.2",
    "@emotion/react": "^11.11.4",
    "@emotion/styled": "^11.11.5",
    "framer-motion": "^11.2.6",
    "react": "^18.2.0",
    "react-dom": "^18.2.0",
    "react-icons": "^5.2.1",
    "react-router-dom": "^6.23.1"
  },
  "devDependencies": {
    "@types/react": "^18.0.28",
    "@types/react-dom": "^18.0.11",
    "@vitejs/plugin-react": "^4.0.0",
    "builder": "^5.0.0",
    "concurrently": "^8.0.1",
    "cross-env": "^7.0.3",
    "electron": "^24.3.1",
    "electron-builder": "^24.13.3",
    "eslint": "^8.38.0",
    "eslint-plugin-react": "^7.32.2",
    "eslint-plugin-react-hooks": "^4.6.0",
    "eslint-plugin-react-refresh": "^0.3.4",
    "vite": "^4.3.2",
    "wait-on": "^7.0.1"
  }
}</code></pre></div><div id="content-10"><img src='https://firebasestorage.googleapis.com/v0/b/personal-blog-darmajr.appspot.com/o/blog-content%2FModash-Display.png?alt=media&token=3b3d3eef-37e4-4c37-91a1-80394b91069e'/></div><div id="content-11"><p>You need to install your production applicaiton and once installed you can look the first page will display the login page. You can modify this style as you like. but the default style is just like it. Enjoy Use this template and you can build this with fast. and promises production environment ?.</p></div>

In the world of application development, Electron.js has become a popular framework for building desktop applications using web technologies. In this blog, we will discuss how to create a desktop application template with Electron.js using React and Vite as the frontend framework, and IndexedDB as the default client-side database.

Create Dashboard app with Electron.js

<div id="content-2"><p>Elasticsearch is a powerful search and analytics engine that is often used in conjunction with various architectural patterns, including CQRS (Command Query Responsibility Segregation). Here’s an overview of why Elasticsearch can be particularly useful in the context of CQRS:</p><h2><strong>Why Use Elasticsearch with CQRS Architecture?</strong></h2><h3><strong>1. Separation of Concerns</strong></h3><p>In CQRS, the system is divided into two parts:</p><ul><li><strong>Command Side</strong>: Handles writes and updates to the system.</li><li><strong>Query Side</strong>: Handles reads and queries from the system.</li></ul><p>Elasticsearch fits naturally into the Query Side of CQRS. It’s designed for fast searches and complex queries, which aligns perfectly with the need for efficient read operations in CQRS. By using Elasticsearch, you can offload complex search and analytics queries from your primary database, allowing it to focus on handling writes and updates.</p><h3><strong>2. Scalability</strong></h3><p>Elasticsearch is built to handle large volumes of data and high query loads. This makes it an excellent choice for the Query Side in CQRS where you might need to execute complex searches, aggregations, and filtering on large datasets. Elasticsearch’s distributed nature allows it to scale horizontally, handling increasing amounts of read traffic without a significant performance hit.</p><h3><strong>3. High Performance</strong></h3><p>Elasticsearch is optimized for search and analytics, providing low-latency responses even for complex queries. It uses inverted indexing to quickly retrieve relevant documents, which is beneficial for applications requiring fast, real-time search capabilities. This performance advantage is critical in the Query Side of CQRS, where read operations need to be efficient and responsive.</p><h3><strong>4. Flexibility in Querying</strong></h3><p>Elasticsearch supports a wide range of query types, including full-text search, filtering, faceting, and aggregations. This flexibility allows you to perform complex and nuanced queries that are often needed for analytics and reporting. In CQRS, where the Query Side might need to provide various views and insights from the data, Elasticsearch can cater to these diverse querying needs.</p></div><div id="content-3"><img style='width:720px;' src='https://firebasestorage.googleapis.com/v0/b/personal-blog-darmajr.appspot.com/o/blog-content%2Fcqrs.png?alt=media&token=6eab7b0b-37d8-49f2-9137-27dadd766c96'/></div><div id="content-4"><h1><strong>Elasticsearch Configuration</strong></h1><p>By default, Elasticsearch comes with a feature called X-Pack, which is an additional plugin provided by Elastic. However, since X-Pack is not open-source, you can disable it if it's not needed. All Elasticsearch configurations can be found in the <code>config/elasticsearch.yml</code> file. Elasticsearch uses YAML format for its configurations, making it easy to manage.</p></div><div id="content-5"><h1><strong>Running Elasticsearch</strong></h1><p>To run Elasticsearch, simply open a terminal and execute the following command:</p></div><div id="content-6"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">./bin/elasticsearch</code></pre></div><div id="content-7"><p>Once it's running, Elasticsearch will be accessible on port 9200 according to the <code>http.port</code> setting in the configuration file. To stop the Elasticsearch application, you can use the <code>Ctrl + C</code> key combination.</p></div><div id="content-9"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">PUT /index_name
</code></pre></div><div id="content-10"><p>The rules for an index name are: it must be lowercase, cannot contain special characters except for <code>-</code>, <code>+</code>, and <code>_</code> (and these cannot be at the beginning), and it cannot exceed 255 bytes.</p></div><div id="content-11"><h2><strong>Elasticsearch Client</strong></h2><p>Elasticsearch communicates using RESTful API, which means we can use HTTP to interact with it. This makes Elasticsearch very flexible and easy to learn, as well as easy to integrate with other applications.</p><h2><strong>Flexible Schema</strong></h2><p>Unlike relational databases, Elasticsearch allows you to insert data into an index without needing to define the schema first. This schema is very flexible, but once it's established, you cannot change the data type of existing fields; you can only add new attributes. For example, if you create an <code>age</code> attribute with a <code>number</code> type, you cannot change it to a <code>string</code> type later on.</p><h2><strong>Primary Key in Elasticsearch</strong></h2><p>When creating a document in Elasticsearch, you are required to include a primary key or ID. Unlike relational databases, in Elasticsearch, the primary key must use the <code>_id</code> field and can only consist of a single field with a string type.</p><h2><strong>Interacting with Elasticsearch Using API</strong></h2><h3>Creating an Index</h3><p>In Elasticsearch, there is no concept of a database like in RDBMS. You can directly create an index (similar to a table in a database). A common practice is to use the application name as a prefix for the index name, such as <code>myapp_users</code>. This prevents index name conflicts when Elasticsearch is used for multiple applications.</p></div><div id="content-12"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">PUT /index_name
</code></pre></div><div id="content-13"><p>The rules for an index name are: it must be lowercase, cannot contain special characters except for <code>-</code>, <code>+</code>, and <code>_</code> (and these cannot be at the beginning), and it cannot exceed 255 bytes.</p><h3>Deleting an Index</h3><p>To delete an index, you can use the DELETE HTTP method. Deleting an index will automatically remove all data within it.</p></div><div id="content-14"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">DELETE /index_name
</code></pre></div><div id="content-15"><h3>Adding Data with Create API</h3><p>To add data to Elasticsearch, you can use the Create API. This API is safe, meaning if the document with the specified <code>_id</code> doesn't exist, it will be saved as a new document. However, if it already exists, a conflict error will occur.</p></div><div id="content-16"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">POST /index_name/_create/id</code></pre></div><div id="content-17"><h3>Retrieving Data with Get API</h3><p>After saving data, you can retrieve it using the Get API. This API returns the data along with its metadata, such as <code>_id</code>, index name, document version, etc.</p></div><div id="content-18"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">GET /index_name/_doc/id
</code></pre></div><div id="content-19"><h3>Multi Get API</h3><p>Elasticsearch also provides a Multi Get API to retrieve multiple documents at once. This is useful when you need to fetch data from multiple indexes in a single API call.</p></div><div id="content-20"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">POST /_mget
</code></pre></div><div id="content-21"><h3>Searching Data with Search API</h3><p>To search for documents without using <code>_id</code>, you can use the Search API. This is a very powerful and complex API that allows you to perform highly specific queries.</p></div><div id="content-22"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">POST /index_name/_search
</code></pre></div><div id="content-23"><h3>Pagination and Sorting</h3><p>The Search API also supports pagination with the <code>from</code> parameter to specify the starting document and the <code>size</code> parameter to specify the number of documents in the response. Additionally, you can sort the search results using the <code>sort</code> parameter.</p></div><div id="content-24"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">POST /index_name/_search
{
  "from": 0,
  "size": 10,
  "sort": [
    { "field_name": "asc" }
  ]
}
</code></pre></div><div id="content-25"><h1><strong>Conclusion</strong></h1><p>Elasticsearch is a powerful and flexible tool for search and data analysis. With a basic understanding of configuration, running the application, and interacting with it using APIs, you can start leveraging Elasticsearch for various use cases such as full-text search, log management, and more. In the next article, we'll dive deeper into optimization and advanced features in Elasticsearch.</p></div>

Elasticsearch is a powerful and scalable search engine built on Apache Lucene, commonly used for full-text search, data analysis, and log management.

Introduction to Elastic Search

<div id="content-3"><h1>Introduction</h1><p>In this article, we will explore how to integrate MySQL with a Flask application using the&nbsp;<code>flask_mysqldb</code>&nbsp;library. This integration allows us to efficiently manage database operations within our Flask web applications. In this lab we will asumme that we are already know to installed the flask application. and we will skip ahead and jump to the sql connection.</p><p><br></p><p>There is a Prerequisites to running this operations:</p><p>Before starting, ensure you have the following installed:</p><ul><li>Python (3.x recommended)</li><li>Flask</li><li>MySQL Server</li><li><code>flask_mysqldb</code>&nbsp;library (<code>pip install flask-mysqldb</code>)</li></ul><p>If we already fulfilled the prerequisites, we will going to install the mysql to the flask application:</p></div><div id="content-4"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">pip install flask-mysqldb</code></pre></div><div id="content-5"><p>Once the installation is completed, we will initiate all the MySql configuration. in this lab we will going to initiate the configurations file trough the <strong><u>app.py</u></strong> file. This is not recommended if the application continue grows to large scale. but for this demonstration, we are going to do that.</p></div><div id="content-11"><h1>Configure the MYSQL Connections</h1></div><div id="content-12"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">from flask import Flask, request, jsonify
from flask_mysqldb import MySQL

app = Flask(__name__)
app.config['MYSQL_HOST'] = 'localhost'
app.config['MYSQL_USER'] = 'username'
app.config['MYSQL_PASSWORD'] = 'password'
app.config['MYSQL_DB'] = 'database_name'

mysql = MySQL(app)</code></pre></div><div id="content-14"><p>in the app.py, im importing the flask_mysqldb that we are already installed so we can use it in the app.py file later. after import the module. initiate the mysql configuration just like above. fill the neccessary field such as the password for db, the host user, etc. depends non your localhost mysql server.</p><p>it will be good if we try to running the flask server application and there is no anomaly when we are running it. when there is nothing error when we are running the flask server. that means that out mysql connection configurations form the flask is successful. and we will continue to do some basic mysql operations in this flask server.</p></div><div id="content-15"><h1>CREATE OPERATIONS</h1></div><div id="content-16"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">app.route('/insert', methods=['POST'])
def insert():
    cur = mysql.connection.cursor()
    data = request.get_json()
    name = data['name']
    age = data['age']
    cur.execute("INSERT INTO users (name, age) VALUES (%s, %s)", (name, age))
    mysql.connection.commit()
    cur.close()
    return jsonify({'message': 'Data inserted successfully'})</code></pre></div><div id="content-17"><p>This code snippet defines a route in a Flask application (<code>/insert</code>) that handles POST requests. When a POST request is made to this endpoint, the function <code>insert()</code> is executed.</p><p>Inside <code>insert()</code>, it establishes a connection to the MySQL database using <code>flask_mysqldb</code>, extracts JSON data containing <code>name</code> and <code>age</code> from the request body, and inserts this data into the <code>users</code> table using an SQL <code>INSERT</code> statement.</p><p>After committing the transaction to the database and closing the cursor, it returns a JSON response confirming successful data insertion. This functionality demonstrates how to integrate MySQL database operations seamlessly into a Flask web application for handling data input.</p></div><div id="content-18"><h1>READ OPERATIONS</h1></div><div id="content-19"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">@app.route('/users', methods=['GET'])
def users():
    cur = mysql.connection.cursor()
    cur.execute("SELECT * FROM users")
    results = cur.fetchall()
    cur.close()
    users_list = []
    for row in results:
        user = {
            'id': row[0],
            'name': row[1],
            'age': row[2]
        }
        users_list.append(user)
    return jsonify({'users': users_list})</code></pre></div><div id="content-21"><p>Let brak down the code piece by piece!</p><p>So This code snippet defines a route in a Flask application (<code>/users</code>) that handles GET requests. When a GET request is made to this endpoint, the function&nbsp;<code>users()</code>&nbsp;is executed.</p><p>Inside&nbsp;<code>users()</code>, it establishes a connection to the MySQL database using&nbsp;<code>flask_mysqldb</code>, executes an SQL&nbsp;<code>SELECT</code>&nbsp;query to fetch all records from the&nbsp;<code>users</code>&nbsp;table, and fetches all results using&nbsp;<code>cur.fetchall()</code>.</p><p>After retrieving the results, it closes the database cursor to release resources. Then, it iterates through the fetched results to construct a list of dictionaries (<code>users_list</code>), where each dictionary represents a user with keys&nbsp;<code>id</code>,&nbsp;<code>name</code>, and&nbsp;<code>age</code>.</p><p>Finally, it returns a JSON response containing the list of users in the format&nbsp;<code>{'users': users_list}</code>. This functionality demonstrates how to retrieve and format data from a MySQL database using Flask, making it accessible via an API endpoint. Very Simple is in it?</p></div><div id="content-22"><h1>Conclusion</h1></div><div id="content-23"><p>By following these steps, you can effectively integrate MySQL into your Flask applications using <code>flask_mysqldb</code>. This setup enables you to perform essential database operations seamlessly within your web development projects.</p></div>

This lab will cover the installation and usage of the flask_mysqldb library. It will include instructions on installing the library, initializing the database in app.py, and providing examples of how to post data and read data from the database.

Integrate Mysql to Flask

<h1>Optimizing Your Content for Top-notch SEO</h1><h2>Introduction</h2><p>SEO (Search Engine Optimization) is a crucial practice in the digital world to enhance the visibility and ranking of your web pages on search engines like Google. By understanding the fundamental principles of SEO, you can increase the likelihood of your content appearing in search results, attract more visitors, and enhance your online presence.</p><h2>1. Understanding Keywords</h2><p>The key to SEO lies in keywords or keyword phrases that potential visitors use when searching for information on search engines. Conduct keyword research to identify terms relevant to your business or topic. Utilize tools like Google Keyword Planner to determine keywords with high search volume.</p><h2>2. High-Quality Content</h2><p>Search engines prioritize content that provides value to users. Ensure your content is informative, relevant, and meets the needs of your audience. Use engaging writing styles and avoid duplicate content. High-quality content not only captivates readers but is also favored by search engines.</p><h2>3. SEO-Friendly URL Structure</h2><p>It's crucial to have URLs that are easily understood by both humans and search engines. Use a clear and descriptive URL structure that is concise and relevant to the content. For example: <code>www.examplewebsite.com/seo-article</code>.</p><h2>4. Heading and Subheading Usage</h2><p>Break down your content into easily digestible sections using headings (H1, H2, H3, etc.) and subheadings. This helps search engines understand the structure and hierarchy of your content. Make sure to include keywords in headings to provide additional signals to search engines.</p><h2>5. Image Optimization</h2><p>Images can enhance the visual appeal of your content but also require SEO optimization. Ensure each image has a descriptive alt attribute containing relevant keywords. Image file sizes should also be minimized to speed up page loading times.</p><h2>6. Quality Backlinks</h2><p>Getting backlinks from high-quality websites can boost your authority and SEO ranking. Build partnerships with other websites in your industry and engage in mutually beneficial link exchanges.</p><h2>Conclusion</h2><p>Optimizing your content for SEO requires attention to detail and a commitment to providing added value to users. By implementing proper SEO practices, you can improve search engine rankings, attract more traffic, and strengthen your online presence.</p>

SEO stands for “search engine optimization.” In simple terms, SEO means the process of improving your website to increase its visibility in Google, Microsoft Bing, and other search engines whenever people search for: Products you sell. Services you provide.

Search Engine Optimization

<h1>React PWA Fundamental</h1><p>A Progressive Web App is a type of web application that combines the best features of web and mobile applications. PWAs can be accessed through a web browser but can also be installed on a user's device like traditional mobile apps. Some key features of PWAs include:</p><ul><li>The ability to work offline</li><li>Fast performance</li><li>Responsive display on various devices</li><li>Access through an icon on the device's home screen</li></ul><p>in this case we will be focused on how to make our react applicaion running on offline, get the requested react module from cache instead from the server. we will be learn about how to cache the customs api fetch that can be store to our react application.</p><h2>What Are Service Workers?</h2><p>Service Workers are a crucial part of PWAs. They are JavaScript files that run in the background and enable features like offline caching, push notifications, and background sync. Service Workers act as intermediaries between your web application and the network, allowing you to control how your PWA behaves in various scenarios.</p><h2>VITE React PWA</h2><p>In Vite React Project, there is some special configuration needed to applied the Progressive Web App. Below is the step-by-step to configure the PWA.</p><h3>Installing vite-plugin-pwa</h3><p>First we need to install the vite-plugin-pwa plugin, just add it to your project as a dev dependency:</p><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">npm install -D vite-plugin-pwa</code></pre><p>Note: to running the implementation of PWA, we need to to build our react vite project, then running the react vite project trough npm run preview. There is some extra configuration to make implementation of PWA running in development mode.</p><h3>Configuring vite-plugin-pwa</h3><p>Edit your vite.config.js / vite.config.ts file and add the vite-plugin-pwa:</p><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">import { defineConfig } from 'vite'
import { VitePWA } from 'vite-plugin-pwa'
import react from '@vitejs/plugin-react'

// https://vitejs.dev/config/
export default defineConfig({
  plugins: [
    react(),
    VitePWA({
      registerType: 'autoUpdate',
      workbox: {
        globPatterns: ['**/*.{js,css,html,ico,png,svg}']
      }
    })
  ],
})</code></pre><h3>Cache External Resources</h3><p>If you have some additional resource like font and css, you must include them into the service workerpre-cache and so your application will work when offline. But in this scenario, we will trying to use some free-api named: https://jsonplaceholder.typicode.com. to fetch the data from it and then stored it to the cache browser. so it can be rendered to a front-end page without the network traffic. The implementation is very easy. we need to addd some property in workbox object named :runtimeCaching. Below the example of how to use it:</p><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;"> runtimeCaching: [
          {
            urlPattern: ({ url }) =&gt; {
              return url.pathname.match('/posts/1')
            },
            handler: 'CacheFirst',
            options: {
              cacheName: 'api-cache',
              cacheableResponse: {
                statuses: [0, 200]
              }
            }
          }
        ]</code></pre><h2>Result Excercise</h2><p>Below is the result example of how the PWA can running the application without the network traffic and requested from the server.</p><img src='https://raw.githubusercontent.com/Barbarpotato/React-PWA-Fundamental/main/git-images/Result.png?token=GHSAT0AAAAAABT2NYLCZWUIB25VOK65BHRIZI6WNAA'/><h2>Service Worker without PWA capabilities</h2><p>Sometimes you don't need the full blown PWA functionality like offline cache and manifest file, but need simple custom Service Worker.</p><h3>Setup the Service Worker</h3><p>You can first check the browsers are supporting the service worker by create the script like below:</p><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">  &lt;script&gt;
    if ('serviceWorker' in navigator) {
      window.addEventListener('load', () =&gt; {
        navigator.serviceWorker.register('/src/serviceWorker.js').then((reg) =&gt; {
          console.log('Worker Registered!')
        }).catch(err =&gt; {
          console.log('Error in service Worker', err)
        })
      })
    }
  &lt;/script&gt;</code></pre><p>This this code is responsible for registering a service worker for your web application.</p><h2>Offline Caching</h2><p>If the services worker is available (it whill show the Worker Registered in your broswer console). Now let's create the serviceWorker.js file in public directory. We can squeeze the serviceWorker file by create some eventlistener that installed some assests from server to Cache Storage. So the client is not calling the resource from the server anymore instead calling from the client browser cache data.</p><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">self.addEventListener('install', (event) =&gt; {
    event.waitUntil(
        caches.open('PWA-Cache').then((caches) =&gt; {
            console.log('Opened Cache')
            return caches.addAll([
                './assets/react.svg',
                '/vite.svg'
            ])
        })
    )
})</code></pre><p>The purpose of this install event handler is to cache these specified assets when the service worker is first installed. Once the assets are cached, they can be served from the cache even if the user is offline, providing offline access to these resources. This is a fundamental step in building Progressive Web Apps (PWAs) that work seamlessly offline.</p><p><br></p><p>After installing all assest from the server to the client. we need to tell the browser that whenever we fetch the data, we need to check the browser cache data first before we calling the server resource. if the client request it is same as the data from a data cache browser, then just use the cache browser data. Below is the example of how the explanation above implemented in javascript:</p><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">self.addEventListener('fetch', (event) =&gt; {
    event.respondWith(
        caches.match(event.request).then((response) =&gt; {
            if (response) {
                // Cache hit, return the response
                return response;
            }
            // Not found in cache, fetch from the network
            return fetch(event.request);
        })
    );
});</code></pre>

A Progressive Web App is a type of web application that combines the best features of web and mobile applications. PWAs can be accessed through a web browser but can also be installed on a user's device like traditional mobile apps.

React PWA Fundamental

<div id="content-0"><h1>What is GraphQL?</h1><p>GraphQL is a query language for your API, and it's designed to request and deliver exactly the data that a client needs. Unlike REST, where the server determines what data is returned, GraphQL puts the control in the hands of the client. This allows for more efficient data retrieval, reduces over-fetching and under-fetching, and enables clients to request multiple resources in a single query.</p><h1>How Does GraphQL Work?</h1><p>GraphQL is based on a strong schema that defines the types and operations available in your API. Clients can request data by specifying what they need, and the server responds with only the requested data in a structured format (typically JSON). The server resolves the query by matching it to the available types and fields in the schema.</p><h1>Setting Up GraphQL with Node.js, Express, and graphql-express</h1><p>This Content will guide you through the process of setting up a GraphQL server on the backend using Node.js, Express, and the graphql-express package. GraphQL is a powerful query language that allows you to request and deliver data with precision, and this setup will enable you to create a flexible API.</p><h1>Prerequisites</h1><p>Before you get started, ensure you have the following prerequisites:</p><ol><li>Node.js installed on your machine.</li><li>A basic understanding of JavaScript and Express.</li></ol><h1>Basic Installation &amp; Usage</h1></div><div id="content-2"><ul><li>Create a New Node.js Project: If you don't already have a Node.js project, create a new directory for your project and run npm init to initialize a new Node.js project.</li></ul></div><div id="content-3"><ul><li>Install Dependencies: You'll need to install the required packages using npm. Run the following command to install Express, GraphQL, and graphql-express:</li></ul></div><div id="content-4"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">npm install express graphql express-graphql</code></pre></div><div id="content-5"><ul><li>Create a Server File: Create a new JavaScript file (e.g., app.js) in your project directory.</li><li>Import Dependencies: In server.js, import the necessary packages and set up the Express app.</li></ul></div><div id="content-6"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">const express = require('express');
const { graphqlHTTP } = require('express-graphql');
const { buildSchema } = require('graphql');

const app = express();</code></pre></div><div id="content-8"><p>Create Schema Folder and create schema.js file. In Schema file there is some component that you need to understand:</p><h2><strong>Create GraphQL Object Type</strong></h2><p>Object Type is a fundamental building block used to define the structure of the data that can be queried from a GraphQL API. It represents a type of object that can be retrieved or manipulated through the API. Object Types play a crucial role in modeling the data and defining the shape of the response that clients can request. Below is the example of how to make Object Type of GraphQL:</p></div><div id="content-9"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">const BookType = new GraphQLObjectType({
    name: 'book',
    fields: () =&gt; ({
        id: { type: GraphQLString },
        name: { type: GraphQLString },
        genre: { type: GraphQLString }
    })
});</code></pre></div><div id="content-10"><h2>Create Entry Point to the GraphQL Schema</h2><p>Query is an operation type in GraphQL used to read or retrieve data from the server. They are created using the RootQuery Object Type and include fields that can be accessed by clients. Below the example code:</p></div><div id="content-11"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">const RootQuery = new GraphQLObjectType({
    name: 'RootQueryType',
    fields: {
        book: {
            type: BookType,
            args: { id: { type: GraphQLString } },
            resolve(parent, args) {
                // code to get data from db
                return books.filter(object =&gt; object.id === args.id)[0]
            }
        }
    }
});</code></pre></div><div id="content-13"><p>This code is defining a RootQuery type in GraphQL, which serves as the entry point for querying data in the schema.</p><ol><li>const RootQuery = new GraphQLObjectType({ ... }): This line creates a new GraphQL Object Type called RootQuery. The RootQuery type is special in GraphQL, as it is the starting point for all read (query) operations. It defines the fields that clients can query from the root of the schema.</li><li>name: 'RootQueryType': This sets the name for the RootQueryType. In this case, it's named "RootQueryType."</li><li>fields: { ... }: Here, you define the available fields within the RootQuery. Each field represents a possible query that clients can make.</li><li>book: { ... }: This defines a field called "book" within the RootQuery. Clients can use this field to query information about books.</li><li>type: BookType: The type field specifies the data type that will be returned by the "book" query. In this case, it's set to the BookType, indicating that when a client queries "book," they will receive data structured according to the BookType.</li><li>args: { id: { type: GraphQLString } }: The args field specifies the arguments that can be provided with the "book" query. In this case, there's one argument named "id," which is of type GraphQLString. It means that clients need to provide an "id" when querying for a book.</li><li>resolve(parent, args) { ... }: The resolve function is where you specify how to fetch the actual data when a client makes a query for "book."</li><li>return books.filter(object =&gt; object.id === args.id)[0]: Within the resolve function, you see code to fetch the data. In this case, it's looking through an array called "books" to find a book with an "id" that matches the one provided by the client. The filter method is used to find the matching book, and [0] is added to return the first matching result. This result will be returned to the client in the shape of a BookType.</li></ol><p>Finally, the code exports a new GraphQLSchema instance with the RootQuery as the query root. This makes the book query available for use in your GraphQL API.</p></div><div id="content-14"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">module.exports = new GraphQLSchema({
query: RootQuery
});</code></pre></div><div id="content-15"><ul><li>Set Up GraphQL Middleware to your app.js: Use the graphqlHTTP middleware to create a GraphQL endpoint for your Express app.</li></ul></div><div id="content-16"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">app.use('/graphql', graphqlHTTP({
    schema: schema,
    graphiql: true
}))</code></pre></div><div id="content-17"><p>by using the graphiql property to your middleware, the backend service will provide the graphql development interface that can be access for demo and simulate accessing the different variant of our http request.</p></div><div id="content-18"><ul><li>Start the Server: Start the Express server on a port of your choice.</li></ul></div><div id="content-19"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">app.listen(4000, () =&gt; {
    console.log(`Now Listening on Post 4000`)
})</code></pre></div><div id="content-20"><ul><li>Testing Your Queries in GrapiQL: You can access the /graphql endpoint in your browser, and the interface will be like this:</li></ul></div><div id="content-21"><img src='https://github.com/Barbarpotato/GraphQL-Fundamental/raw/main/images/graphiql-example1.png'/></div><div id="content-22"><h2>List Type</h2><p>So far we have already built the relationship between the book and who is the author from the book. Now we want to build the relationship between the author and the book. we want to know if some author are called trough the request, we want to know what books they are created. in other words we call it&nbsp;<code style="background-color: var(--bgColor-neutral-muted, var(--color-neutral-muted));">one to many</code>&nbsp;relationship if we are on the relational database environment, which is one author can have many books they created. Below is the example of how to implement it in GraphQL:</p></div><div id="content-23"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">// we added additional object for the one to many relationship display purposes.
const books = [
    { name: 'Name of thw Wind', genre: 'Fantasy', id: '1', authorId: '1' },
    { name: 'The Final Empire', genre: 'Fantasy', id: '2', authorId: '2' },
    { name: 'The Long Earth', genre: 'Sci-Fi', id: '3', authorId: '3' },
    { name: 'The Hero Of Ages', genre: 'Fantasy', id: '4', authorId: '2' },
    { name: 'The Colourof Magic', genre: 'Fantasy', id: '5', authorId: '3' },
    { name: 'The Loght Fantastic', genre: 'Fantasy', id: '6', authorId: '3' }
];

const authors = [
    { name: 'Patrick Bateman', age: 29, id: '1' },
    { name: 'Bruce Wayne', age: 33, id: '2' },
    { name: 'Peter Parker', age: 25, id: '3' }
]

const AuthorType = new GraphQLObjectType({
    name: 'author',
    fields: () =&gt; ({
        id: { type: GraphQLID },
        name: { type: GraphQLString },
        age: { type: GraphQLInt },
        book: {
            type: new GraphQLList(BookType),
            resolve(parent, args) {
                return books.filter(object =&gt; object.authorId === parent.id)
            }
        }
    })
});</code></pre></div><div id="content-24"><ul><li>In above code, dont forget that we are going to return multiple object from the book fields, which we are need the&nbsp;<code style="background-color: var(--bgColor-neutral-muted, var(--color-neutral-muted));">GraphQLList</code>&nbsp;imported from the&nbsp;<code style="background-color: var(--bgColor-neutral-muted, var(--color-neutral-muted));">graphql</code>&nbsp;instance.</li><li>we are returning the processed data from the authorType. This data will be processed again in resolve function book field.</li><li>the result query if we success build this one to many relations:</li></ul></div><div id="content-25"><img src='https://github.com/Barbarpotato/GraphQL-Fundamental/raw/main/images/type-list.png'/></div><div id="content-26"><h2>All Objects</h2><p>For some cases, we need to return all list of books, or all list of author that we want to the client. To do this we just added some field in the&nbsp;<code style="background-color: var(--bgColor-neutral-muted, var(--color-neutral-muted));">RootQuery</code>&nbsp;of our Schema:</p></div><div id="content-27"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">const RootQuery = new GraphQLObjectType({
    name: 'RootQueryType',
    fields: {
        ...,
        ...,
        books: {
            type: new GraphQLList(BookType),
            resolve(_parent, _args) {
                return books
            }
        },
        authors: {
            type: new GraphQLList(AuthorType),
            resolve(_parent, _args) {
                return authors
            }
        }
    }
});</code></pre></div><div id="content-29"><p>The result output from the graphiql will be like this:</p></div><div id="content-30"><img src='https://github.com/Barbarpotato/GraphQL-Fundamental/raw/main/images/all-objects.png'/></div>

GraphQL is a powerful query language for your API, and it provides a more efficient, powerful, and flexible alternative to the traditional REST API.

GraphQL Fundamental

<div id="content-0"><h1><strong>Real-World Case</strong></h1></div><div id="content-1"><img style='width:720px;' src='https://raw.githubusercontent.com/Barbarpotato/React-Lazy-Load/main/git-image/Scenario.png'/></div><div id="content-2"><p>When building a dashboard application. This application has high complexity and requires a large size to be loaded by application users in the future. For example, the dashboard application has several different features, depending on the position of each application user. for example, users with the admin position have a different features in the application compared to users with other positions, for example Sales. When Sales enters a dashboard application. Of course, the features contained in admin will not appear for users with sales positions, BUT users with sales positions will still load the features contained in admin even though they are not used. Vice Versa, This affects the performance, speed and efficiency of an application running.</p><p><br></p><h2><strong>Solution</strong></h2><p>To asnwer the prbolem above, we will be use&nbsp;<code style="color: rgb(248, 248, 242); background-color: var(--bgColor-neutral-muted, var(--color-neutral-muted));">Lazy Load</code>&nbsp;technique, you can optimize the initial loading time and improve the performance of your React application by dynamically loading components as needed.</p><h2><strong>Usage</strong></h2><p>Here are the steps on how to use React Lazy Load in your project:</p><h3><strong>Create the routes for the web page.</strong></h3><p>We have 3 different pags for this case, Main page which can be access for all the role position (Admin, Sales). we have the sales component which can be access by sales person, and the last page is the admin page, where it can only be access by the admin person.</p></div><div id="content-3"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">import { Routes, Route } from 'react-router-dom'
import Sales from './pages/Sales'
import Home from './pages/Home'
import Admin from './pages/Admin'
import './App.css'

function App() {

  return (
    &lt;Routes&gt;
      &lt;Route index element={&lt;Home /&gt;} /&gt;
      &lt;Route path='/sales' element={&lt;Sales /&gt;} /&gt;
      &lt;Route path='/admin' element={&lt;Admin /&gt;} /&gt;
    &lt;/Routes&gt;
  )
}

export default App</code></pre></div><div id="content-6"><p>If we dont implement the lazy load technique&nbsp;&nbsp;our react application&nbsp;it will be load&nbsp;&nbsp;three pages&nbsp;which are Main, Sales&nbsp;&nbsp;the Admin page. If we implement the lazy load, whenever we access the Sales Page, it will be load the Main&nbsp;&nbsp;the Sales Page, Vice Versa</p><h2>Applied the Lazy Load to your React App</h2><p>After build the different page&nbsp;&nbsp;route, we can implement the lazy load by using the lazy&nbsp;&nbsp;Suspense</p></div><div id="content-7"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">import { Routes, Route } from 'react-router-dom'
import { lazy, Suspense } from 'react'
import Home from './pages/Home'
import './App.css'

const Admin = lazy(() =&gt; import('./pages/Admin'))
const Sales = lazy(() =&gt; import('./pages/Sales'))

function App() {

  return (
    &lt;Routes&gt;

      &lt;Route index element={&lt;Home /&gt;} /&gt;

      &lt;Route path='/sales' element={
        &lt;Suspense fallback={&lt;div&gt;Loading Content...&lt;/div&gt;}&gt;
          &lt;Sales /&gt;
        &lt;/Suspense&gt;} /&gt;

      &lt;Route path='/admin' element={
        &lt;Suspense fallback={&lt;div&gt;Loading Content...&lt;/div&gt;}&gt;
          &lt;Admin /&gt;
        &lt;/Suspense&gt;} /&gt;

    &lt;/Routes &gt;
  )
}

export default App</code></pre></div><div id="content-8"><h2><strong>Implementation Result</strong></h2><p>Now, if you go to the Dev Console -&gt; Source. You need to check the page source file. and then check the src folder -&gt; pages . We cannot see our Admin and Sales Component. If we accessing the admin route or the sales route, it will appear in the source dev console. which means that our lazy load implement successfully.</p></div><div id="content-10"><img style='width:720px;' src='https://raw.githubusercontent.com/Barbarpotato/React-Lazy-Load/main/git-image/lazy-load.png'/></div>

React Lazy Load is a technique used to load React components lazily, i.e., only when they are needed. This helps reduce the initial bundle size and improve the performance of your React application. You can easily implement lazy loading in your React project using React.lazy() along with Suspense.

React Lazy Load

<div id="content-0"><h1>What is ETL?</h1><p><strong>ETL</strong> stands for <strong>Extract, Transform, Load</strong>. It is a crucial process in data management and integration, typically used in data warehousing and data analytics. Here's a brief overview:</p><ul><li><strong>Extract</strong>:</li><li class="ql-indent-1">This step involves retrieving data from various sources such as databases, files, APIs, or other systems. The goal is to gather raw data for further processing.</li><li><strong>Transform</strong>:</li><li class="ql-indent-1">Once the data is extracted, it often needs to be cleaned, formatted, and transformed to fit the desired structure or schema. This step may include filtering, sorting, aggregating, and converting data to ensure consistency and compatibility.</li><li><strong>Load</strong>:</li><li class="ql-indent-1">The final step is to load the transformed data into a target system, such as a database, data warehouse, or data lake. This makes the data available for querying, analysis, and reporting.</li></ul><p><br></p><h1>ETL using Shell Scripts</h1><p>In this lab we will covering about read and extract data from various delimited thing using shell command. We'll understand how to handle file input, parse the data, and prepare it for further processing. Master the techniques for transforming text data using powerful shell utilities like tr command. and finally load the data to the database. in this lab we will use PostgreSQL. </p></div><div id="content-1"><h1><strong>Exercise 1 - Extracting data using 'cut' command</strong></h1><h2>Extracting characters</h2><p></p></div><div id="content-2"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">echo "database" | cut -c1-4</code></pre></div><div id="content-4"><p>You should get the string ‘data’ as output.</p><p>The command below shows how to extract 5th to 8th characters.</p></div><div id="content-5"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">echo "database" | cut -c5-8</code></pre></div><div id="content-6"><p>You should get the string ‘base’ as output.</p><p>Non-contiguous characters can be extracted using the comma.</p></div><div id="content-7"><h2>Extracting fields/columns</h2><p>We can extract a specific column/field from a delimited text file, by mentioning</p><ul><li>the delimiter using the&nbsp;<code style="background-color: transparent; color: rgb(255, 153, 0);">-d</code>&nbsp;option, or</li><li>the field number using the&nbsp;<code style="background-color: transparent; color: rgb(255, 153, 0);">-f</code>&nbsp;option.</li></ul><p>The /etc/passwd is a “:” delimited file.</p><p>The command below extracts usernames (the first field) from /etc/passwd.</p></div><div id="content-8"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">cut -d":" -f1 /etc/passwd</code></pre></div><div id="content-9"><p></p></div><div id="content-10"><img  style='width:600px; height:100%'src='https://firebasestorage.googleapis.com/v0/b/personal-blog-darmajr.appspot.com/o/blog-content%2Fetl_using_shell_script_img_1.png?alt=media&token=7a3e427e-002d-4d12-875f-8f8160cc984d'/></div><div id="content-11"><h1><strong>Exercise 2 - Transforming data using 'tr'</strong></h1></div><div id="content-12"><h2>Translate from one character set to another</h2><p>The command below translates all lower case alphabets to upper case.</p></div><div id="content-13"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">echo "Shell Scripting" | tr "[a-z]" "[A-Z]" echo "Shell Scripting" | tr "[:lower:]" "[:upper:]" echo "Shell Scripting" | tr "[A-Z]" "[a-z]"</code></pre></div><div id="content-14"><h1>Prepared the Database for the practice</h1><p>In this exercise we will create a table called&nbsp;<code style="background-color: transparent; color: rgb(255, 153, 0);">users</code>&nbsp;in the PostgreSQL database using PostgresSQL CLI. This table will hold the user account information.</p><p>The table&nbsp;<code style="background-color: transparent; color: rgb(255, 153, 0);">users</code>&nbsp;will have the following columns: uname, uid, home</p></div><div id="content-15"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">create table users(username varchar(50),userid int,homedirectory varchar(100));</code></pre></div><div id="content-17"><h1>Create the Shell Script File</h1>touch csv2db.sh. Then Open the file in the editor. Copy and paste the following lines into the newly created file.</p></div><div id="content-18"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;"># This script # Extracts data from /etc/passwd file into a CSV file. # The csv data file contains the user name, user id and # home directory of each user account defined in /etc/passwd # Transforms the text delimiter from ":" to ",". # Loads the data from the CSV file into a table in PostgreSQL database.</code></pre></div><div id="content-19"><p>We need to add lines of code to the script that will xtract user name (field 1), user id (field 3), and home directory path (field 6) from /etc/passwd file using the&nbsp;<code style="background-color: transparent; color: rgb(255, 153, 0);">cut</code>&nbsp;command. </p></div><div id="content-20"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">cut -d":" -f1,3,6 /etc/passwd</code></pre></div><div id="content-21"><p>then run the script. -&gt; bash csv2db.sh</p></div><div id="content-22"><p>let us change the last line of command so it not directly printed trough the terminal instead we will save the output to .txt file</p></div><div id="content-23"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">cut -d":" -f1,3,6 /etc/passwd &gt; extracted-data.txt</code></pre></div><div id="content-24"><img style='width:720px; heigth:100%;' src='https://firebasestorage.googleapis.com/v0/b/personal-blog-darmajr.appspot.com/o/blog-content%2Fetl_using_shell_script_img_2.png?alt=media&token=3b31b28f-40bf-4f3e-952a-4d0607d133c6'/></div><div id="content-25"><p>The extracted columns are separated by the original “:” delimiter. You need to convert this into a “,” delimited file. Add the below lines at the end of the script and save the file.</p></div><div id="content-26"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">tr ":" "," &lt; extracted-data.txt &gt; transformed-data.csv</code></pre></div><div id="content-27"><p>Run the script and then it automatically save the .csv file to your system. you can read it from the terimnal by calling this command: cat transformed-data.csv</p></div><div id="content-29"><h3>To load data from a shell script, you will use the&nbsp;<code style="background-color: transparent; color: rgb(255, 153, 0);">psql</code>&nbsp;client utility in a non-interactive manner. This is done by sending the database commands through a command pipeline to&nbsp;<code style="background-color: transparent; color: rgb(255, 153, 0);">psql</code>&nbsp;with the help of&nbsp;<code style="background-color: transparent; color: rgb(255, 153, 0);">echo</code>&nbsp;command.</h3><p><br></p></div><div id="content-30"><p>PostgreSQL command to copy data from a CSV file to a table is&nbsp;<code style="background-color: transparent; color: rgb(255, 153, 0);">COPY</code>.</p><p>The basic structure of the command which we will use in our script is</p></div><div id="content-31"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">COPY table_name FROM 'filename' DELIMITERS 'delimiter_character' FORMAT;</code></pre></div><div id="content-32"><p></p></div><div id="content-33"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">echo "\c template1;\COPY users FROM '/home/project/transformed-data.csv' DELIMITERS ',' CSV;" | psql --username=postgres --host=localhost</code></pre></div><div id="content-34"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">echo '\c template1; \\SELECT * from users;' | psql --username=postgres --host=localhost</code></pre></div><div id="content-35"><img style='width:720px; height:100%;' src='https://firebasestorage.googleapis.com/v0/b/personal-blog-darmajr.appspot.com/o/blog-content%2Fetl_using_shell_script_img_3.png?alt=media&token=ff23b79e-4971-4f41-9060-58b47f009358'/></div>

In this lab, we perform ETL (Extract, Transform, Load) using shell scripts. Learn to extract data from delimited files, transform text data with cut and tr, and load it into a PostgreSQL database. This hands-on approach will gain essential skills for automating data management and integration.

ETL using shell scripts

<div id="content-1"><h2>Big-O notation</h2><p><br></p><p>Evaluating an application's performance ensures that the code written is good and fit for purpose. The question is how do we evaluate efficiency? When we measure electricity, we use kilowatt-hours, which means how many kilowatts an appliance will use if it runs for an hour. The appliance will not always run for an hour, and it may have different requirements depending on the setting used, it is more of a general rule-of-thumb for evaluating cost.</p><p>When evaluating coding solutions, Big-O notation is used. So, Big-O notation is the kilowatt hour of code evaluation. It can be applied to measuring how much time a piece of code will take or how much space it will use in memory. Not all processors will run at the same speed, so instead of timing an application, you count the number of instructions an application initiates.</p><p>Which measurement reflects the quickest possible execution of some code? Let's explore which measurement reflects the quickest possible execution of some code.</p><p><code style="background-color: var(--bgColor-neutral-muted, var(--color-neutral-muted));">O(1)</code>&nbsp;You use a constant time algorithm that takes O(1) (O-of-one) time to compute. This determines that it will only take one computation to complete a task. An example of this is to print an item from an array.</p></div><div id="content-2"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;"># An array with 5 numbers 
array = [0,1,2,3,4]

# retrieve the number found at index location 3 
print(array[3]) </code></pre></div><div id="content-3"><p>In this instance, no matter how many values exist in the array, the approach has a Big-O of one. This means that running this code is considered O(1).</p><p><code style="background-color: var(--bgColor-neutral-muted, var(--color-neutral-muted));">O(n)</code>&nbsp;Next, let's explore an example of O(n). Taking the same array, an if statement is written that looks for the number 5. To establish that 5 is not there, it has to check every item in the array.</p></div><div id="content-4"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;"># An array with 5 numbers 
array = [0,1,2,3,4] 

if 5 in array:
    print("five is alive")</code></pre></div><div id="content-5"><p></p></div><div id="content-7"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;"># an array with 10 numbers 
array = [0,1,2,3,4,6,7,8,9,10]

if 5 in array:
    print("five is still alive")</code></pre></div><div id="content-8"><p><code style="background-color: var(--bgColor-neutral-muted, var(--color-neutral-muted));">O(log n)</code>&nbsp;This search is less intensive than O(n) but more work than O(1). O(log n) is a logarithmic search and it will increase as new inputs are added but these inputs only offer marginal increases. An excellent example of this in action is a binary search. Binary search is covered in more detail later in the course.</p><p>Now, imagine playing a guessing game with the following prompts: too high, too low, or correct. You are given a range of 100 to 1. You may decide to approach the problem systematically. First, you guess 50 – too high. So, you guess 25 – which is too high. You may choose then to go 12 or 13. What is happening here is that you are halving the search space with each guess.</p><p>So, while the input to this function was 100 using a binary search approach, you should come upon the answer in under 5 or 6 guesses. This solution would have a time complexity of O(log n). Even if n (the range of numbers entered) is ten times bigger. It will not take ten times as many guesses.</p><p>Here is a breakdown of those steps on the array.</p></div><div id="content-9"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">array = [0,1,2,3,4,6,7,8,9,10]

print("##Step One")
print("Array")
print(array)
midpoint = int(len(array)/2)
print("the midpoint at step one is: " , array[midpoint])

print()

print("##Step Two")
array = array[:midpoint] # 6 is the midpoint of the array 
print("Array")
print(array)
# running this shows the numbers left to check 
# is 5 &lt; 3 
# no 
# so discard the left hand side 

# so the array is halved again 
midpoint=int(len(array)/2)
print("the midpoint is: ",  array[midpoint])

print()
print("##Step Three") 
array = array[midpoint:] # so the array is halved at the midpoint
print(array)# check for the midpoint 
midpoint=int(len(array)/2)
print("the midpoint is: " , array[midpoint])
# is 4 &lt; 5 
# yes look to the right

print()
print("##Step Four") 
print(array[midpoint:]) 
# check for the midpoint 
array = array[midpoint:] # so the array is halved at the midpoint
midpoint=int(len(array)/2)



print()
print("##Step Five") 
array = array[midpoint:] 
print(array)
print("only one value to check and it is not 5")</code></pre></div><div id="content-10"><p>You will notice that to determine if 5 is present, it took 5 steps. That is a big-O score of O(5). You can see that this is bigger than O(1) but smaller than O(n). Now, what happens when the array is extended to 100? When looking for a number in an array of 10, it took 5 guesses. Looking at an array of 100 will not take 50 guesses; it will take no more than 10. Equally, if the list is extended to 1000, the guesses will only go up to 15-20.</p><p>From this, we can see that it is not O(1) because the answer is not immediate. It is not big-O(n) because the number of guesses does not go up with the size n of the array. So here, one says that the complexity is O(log(n)).</p><p>To gain greater insight into how the log values are only a gradual rise, look at a log table up to 100,000,000. This lens shows that O(log n) incurs only a minimal processing cost. Running a binary search on an array with any n values will, in a worst-case scenario, always make the number of computations found in the log values column.</p><p><code style="background-color: var(--bgColor-neutral-muted, var(--color-neutral-muted));">O(n^2)</code>&nbsp;is heavy on computation. This is quadratic complexity, meaning that the work is doubled for every element in the array. An excellent way to visualize this is to consider that you have a variety of arrays. In keeping with the earlier example, let's explore the following code:</p></div><div id="content-11"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">new_array=[] # an array to hold all of the results 
# array with five numbers 
array = [0,1,2,3,4]
for i in range(len(array)): # the array has five values, so this is n=5 
    for j in range(len(array)): # still the same array so n = 5 
        new_array.append(i*j) # every computation made is stored here 

print(len(new_array)) #how big is this new array ? </code></pre></div><div id="content-12"><p>The first loop wi ll equal the number of elements input, n. The second loop will also look at the number of input elements, n. So, the overall complexity of running this approach can be said to be n*n which is n^2 (n-squared). To find out how many computations were made, you have to print out the number of times n was used in the loop as below.</p><p>If you know that the array has 25 elements, then you understand the principles of calculating Big-O notation. To further test your knowledge, how many computations would be required if n = 6? Meaning the array had 6 values? The answer is 6 x 6 so 36.</p></div><div id="content-13"><h1>Working with Space Complexity</h1><p><br></p><p>Some Algorithms like&nbsp;<code style="background-color: var(--bgColor-neutral-muted, var(--color-neutral-muted));">hash tables</code>&nbsp;provide very fast lookups in&nbsp;<code style="background-color: var(--bgColor-neutral-muted, var(--color-neutral-muted));">O(10) time</code>. However, to work efficiently, they must have a lookup every element stored. This results in a space complexity of O(n). The big O-notation for space complexity is the same as for the time O(1), O(log log n), O(log n) and so on. In all these notations&nbsp;<code style="background-color: var(--bgColor-neutral-muted, var(--color-neutral-muted));">n</code>&nbsp;refers to the size of the input. This is often measured in bytes.</p><p>Different languages have different memory costs associated with them. In java for instance, an integer requires 4 bytes of memory. A blank array will consume 12 bytes for the header object and an additional 4 bytes for padding. Thus, if n refers to an array of integers size 4, then the total memory requirement is 32 bytes of memory.&nbsp;<code style="background-color: var(--bgColor-neutral-muted, var(--color-neutral-muted));">(4 * 4) + 12 + 4 = 32</code></p><p>When discussing space complexity, you have to consider what the increase in input size has on the overall usage. Space Compexity is the total of auxiliary space + input space The space complexity of a problem can be broken into two sections namely auxiliary and input space:</p><ul><li>Auxiliary space is the space required to hold all data required for the solution. It refers to the temporary space needed to compute a given solution.</li><li>Input space refers to the space required to add data to the function, algorithm, application or system that you are evaluating.</li></ul></div><div id="content-14"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">const a = 1;
const b = 10;
const c = 100;
const sum = (a, b, c) =&gt; a + b + c;
const d = sum(a, b, c);</code></pre></div><div id="content-15"><p>In the above function sum, we can resolve the space complexity required by the memory requirement. Given that all arguments in the above example are integers and the return value is an integer of set size, we know this will be constant.</p><p>Given the Number type in JavaScript is 64-bit (8 bytes), we can resolve that the memory requirement for a, b and c is (24). Therefore, the function sum has&nbsp;<code style="background-color: var(--bgColor-neutral-muted, var(--color-neutral-muted));">O(1)</code>&nbsp;constant space complexity given that we know the constant requirement of 24 bytes of data space for this function.</p><p>Looking at an example in C that requires a dynamic amount of memory in an array:</p></div><div id="content-16"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">int sum(int a[], int n) {
    int x = 0;      // 4 bytes for x
    for(int i = 0; i &lt; n; i++)  // 4 bytes for i
    {
        x  = x + a[i];
    }
    return(x);
}</code></pre></div><div id="content-17"><p>Firstly, we know that int types in C require 4 bytes of space. Here we note that 4*n bytes of space is required for the array a[] of integers. The remaining variables x, n, i and the return value each require a constant 4 bytes each of memory give that they are integers.</p><p>This gives us a total memory requirement of (4n + 12). This itself is&nbsp;<code style="background-color: var(--bgColor-neutral-muted, var(--color-neutral-muted));">O(n)</code>&nbsp;linear space complexity since the memory requires linearly increases with input value n.</p><p>What is important to note with this example is that the auxiliary space required for the above sum function is actually&nbsp;<code style="background-color: var(--bgColor-neutral-muted, var(--color-neutral-muted));">O(1)</code>&nbsp;constant given that the auxiliary variables are only x and i which totals a (8) memory requirement (constant).</p></div>

In this reading, We will explore a worked example of a piece of code written in Python, along with how you would evaluate it using Big-O notation.

Working with Time & Space Complexity 

<div id="content-0"><p class="ql-align-justify">Apache Airflow is an open-source platform designed for orchestrating and managing complex workflows and data pipelines. It allows users to programmatically author, schedule, and monitor workflows through a rich web-based user interface. Airflow uses directed acyclic graphs (DAGs) to represent workflows, making it highly flexible and scalable for a wide range of use cases. With built-in support for dynamic pipeline generation, Airflow enables users to create workflows that adapt to changes in data and business logic. Additionally, its extensible architecture allows integration</p></div><div id="content-1"><img src='https://firebasestorage.googleapis.com/v0/b/personal-blog-darmajr.appspot.com/o/blog-content%2Fapache_airflow_architecture.png?alt=media&token=2ff31468-d678-4134-b46c-4f0da4f4ff93'/></div><div id="content-3"><p class="ql-align-justify">Apache Airflow's architecture is designed to be modular and scalable, consisting of several key components: the Scheduler, the Executor, the Web Server, and the Metadata Database. The Scheduler handles the scheduling of tasks, ensuring they run at the correct times and in the correct order. The Executor runs the tasks, which can be distributed across multiple workers for scalability. The Web Server provides a user-friendly interface for managing and monitoring workflows, while the Metadata Database stores the state of tasks and workflows.</p><p class="ql-align-justify">Workflows in Airflow are defined as Directed Acyclic Graphs (DAGs), where each node represents a task, and edges define the dependencies between tasks. This structure allows for clear visualization of the workflow and its dependencies. Tasks within a DAG can range from simple data processing steps to complex data transfer operations, and they can be scheduled to run at specific intervals or triggered by external events. Airflow supports a wide range of operators for different tasks, including BashOperator for running bash scripts, PythonOperator for executing Python code, and many more for interacting with various data systems and services. This flexibility and the ability to create dynamic, code-driven workflows make Airflow a powerful tool for orchestrating data pipelines and workflows.</p><p class="ql-align-justify">In this lab we will learn how to install apache airflow and create some basic DAG to perform ETL task. First we want to make sure our apache airflow in in our local machine for demonstration. We will install using docker and docker compose for it. Lets create new folder and named it as we want. In this case im going to create folder name apache_airflow. Then lets&nbsp;create a Dockerfile inside that folder and fill the file with this code:</p><p><br></p></div><div id="content-4"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">FROM apache/airflow:latest
USER root
RUN apt-get update && \
    apt-get -y install git && \
    apt-get clean

USER airflow
</code></pre></div><div id="content-5"><p class="ql-align-justify">Overall, this Dockerfile snippet customizes the Airflow Docker image by adding Git, which can be useful for tasks such as cloning repositories or managing code directly within the Airflow environment. Lets run this docker file. If the build image is successful, we then jump to the next step where we are going to create a new file docker compose to use our image and create a volums for our data. Below is the example of how we build our compose to run apache airflow:</p></div><div id="content-6"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">version: '3'
services:
  apacheairflow:
    image: apacheairflow:latest

    volumes:
      - ./airflow:/opt/airflow

    ports:
      - "8080:8080"
    command: airflow standalone
</code></pre></div><div id="content-7"><p class="ql-align-justify">After this, we can access our apache airflow in our local machine by accessing trough <a href="http://localhost:8080/" target="_blank">http://localhost:8080</a>. Dont forget to build up our compose file. By running docker-compose up -d. We can type our username Admin and we wil get our password from our folder generated from /airflow/stand_alone_admin_password.txt.</p><p class="ql-align-justify">Now we can create our DAG file in our /airflow folder. Create new folder name dags and fill the python file inside of it. The name is our wishes.&nbsp;Then we can create a simple task from it. But we need to import some thing like:</p></div><div id="content-8"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;"># Import the libraries
from datetime import timedelta
# The DAG object; we'll need this to instantiate a DAG
from airflow.models import DAG
# Operators; you need this to write tasks!
from airflow.operators.python import PythonOperator

# This makes scheduling easy
from airflow.utils.dates import days_ago</code></pre></div><div id="content-9"><p class="ql-align-justify">Now we can define our function, DAG Argument, DAG Definitions, Define the task, and create a task pipeline from this file.</p><p class="ql-align-justify">In this case. I just want to grab some data from the api and save it to json file.Lets create our functions first called extract here is the code for the extract data from the api random jokes:</p></div><div id="content-10"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">def extract():
    response = requests.get('https://official-joke-api.appspot.com/random_joke')
    joke = response.json()

    # Define the path to the JSON file
    file_path = '/opt/airflow/jokes/joke.json'
    
    try:
        # Ensure the directory exists
        os.makedirs(os.path.dirname(file_path), exist_ok=True)
        
        # Read existing jokes from the JSON file if it exists
        if os.path.exists(file_path):
            with open(file_path, 'r') as f:
                # Try to load the existing jokes, handle if file is empty
                content = f.read()
                jokes = json.loads(content) if content else []
        else:
            jokes = []

        # Append the new joke to the list
        jokes.append(joke)
        
        # Write the updated list of jokes to the JSON file
        with open(file_path, 'w') as f:
            json.dump(jokes, f, indent=4)
        
        print(f"Joke saved to {file_path}: {joke['setup']} - {joke['punchline']}")
    except Exception as e:
        print(f"Error: {e}")
        raise</code></pre></div><div id="content-11"><p>After that we can create our DAG Arguments and DAG Definitions:</p></div><div id="content-12"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;"># You can override them on a per-task basis during operator initialization
default_args = {
    'owner': 'Darmawan',
    'start_date': days_ago(0),
    'email': ['darmawanjr88@gmail.com'],
    'retries': 3,
    'retry_delay': timedelta(minutes=5),
}

# Define the DAG
dag = DAG(
    'jokes-callable-dag',
    default_args=default_args,
    description='My first DAG',
    schedule_interval=timedelta(days=1),
)
</code></pre></div><div id="content-13"><p>Then we can define the task named execute_extract to call the extract function</p></div><div id="content-14"><p>And build the the task pipeline for now we just have a single task when we have multiple task we can call our defined task like a sequence call like task_1 &gt;&gt; task_2 &gt;&gt; task_3 and so on. Now we can go to the web UI and triggering the DAG that we build.</p></div><div id="content-15"><img src='https://firebasestorage.googleapis.com/v0/b/personal-blog-darmajr.appspot.com/o/blog-content%2Fapache_airflow_1.png?alt=media&token=1b05157a-20dc-4748-bb40-ff9bbaeb23be'/></div><div id="content-16"><p>Horray we dit it! We can create a simple task from the apache airflow. We can make another complex task from our idea and our problems that we faced from our life.</p></div>

In this lab, you will explore the Apache Airflow web user interface (UI). You will then create a Direct Acyclic Graph (DAG) using PythonOperator and finally run it through the Airflow web UI.

Create a DAG for Apache Airflow with Python Operator

<div id="content-0"><h1 class="ql-align-justify"><strong>Introduction</strong></h1><p class="ql-align-justify">Apache Kafka is an event streaming platform that helps in moving and storing large amounts of data in real-time. It is like a central hub where different sources of data can send their events, and these events can be consumed by various applications or systems.</p><p class="ql-align-justify">Think of it as a postal service. Imagine you have different people sending letters from different locations, and you want all these letters to be collected in one place so that anyone who needs them can access them. Apache Kafka acts as that central place where all the letters (events) are collected and stored. It ensures that the events are delivered reliably and can be accessed by multiple applications or systems.</p><p class="ql-align-justify">Kafka is highly scalable, meaning it can handle a large volume of data and process it quickly. It is also fault-tolerant, which means it can recover from failures and ensure that no data is lost. Additionally, Kafka is open source, so it can be used for free and customized according to specific needs.</p></div><div id="content-1"><h1 class="ql-align-justify"><strong>Architecture</strong></h1></div><div id="content-2"><img style='width:720px;' src='https://firebasestorage.googleapis.com/v0/b/personal-blog-darmajr.appspot.com/o/blog-content%2Fapache_kafka_architecture.png?alt=media&token=4f637bf3-00ec-4dc6-9fb9-9f5602e9d18a'/></div><div id="content-3"><p class="ql-align-justify">The architecture of Apache Kafka consists of several key components that work together to enable the efficient streaming and processing of data. Here is a simplified explanation of the architecture and its components:</p><ol><li class="ql-align-justify">Producers: Producers are the sources of data in Kafka. They send events or messages to Kafka topics. Topics are like categories or channels where events are organized and stored.</li><li class="ql-align-justify">Topics: Topics are the central entities in Kafka. They represent a specific category or stream of events. Producers send events to specific topics, and consumers subscribe to topics to receive those events.</li><li class="ql-align-justify">Brokers: Brokers are the servers in the Kafka cluster. They receive events from producers, store them, and distribute them to consumers. Each broker can handle a specific amount of data and provides fault tolerance by replicating data across multiple brokers.</li><li class="ql-align-justify">Consumers: Consumers are the applications or systems that subscribe to topics and receive events from Kafka. They process the events according to their specific requirements.</li><li class="ql-align-justify">Partitions: Topics are divided into partitions, which are individual ordered sequences of events. Each partition is stored on a specific broker. Partitioning allows for parallel processing and scalability.</li><li class="ql-align-justify">Consumer Groups: Consumer groups are a way to scale the consumption of events. Multiple consumers can be part of a consumer group, and each consumer within the group will receive a subset of the events from the topic partitions.</li><li class="ql-align-justify">ZooKeeper (deprecated in newer versions): ZooKeeper is a distributed coordination service that was used in older versions of Kafka for managing the cluster and maintaining metadata. However, in newer versions, Kafka Raft (KRaft) is used to eliminate the dependency on ZooKeeper.</li><li class="ql-align-justify">Kafka Connect: Kafka Connect is a framework for importing and exporting data to and from Kafka. It allows seamless integration with external systems and data sources.</li><li class="ql-align-justify">Kafka Streams: Kafka Streams is a library that enables stream processing of data within Kafka. It allows developers to build real-time applications and perform transformations, aggregations, and analytics on the data.</li></ol></div><div id="content-4"><h1 class="ql-align-justify"><strong>Core Component</strong></h1><p class="ql-align-justify">Here's a breakdown of the core components of Kafka:</p><p class="ql-align-justify">1. Brokers: These are dedicated servers that receive, store, process, and distribute events. They are like the central hub for all the events.</p><p class="ql-align-justify">2. Topics: These are containers or databases of events. Each topic stores specific types of events, such as logs, transactions, or metrics.</p><p class="ql-align-justify">3. Partitions: Topics are divided into different partitions, which are like smaller sections within a topic. This helps with scalability and performance.</p><p class="ql-align-justify">4. Replications: Partitions are duplicated and stored in different brokers. This ensures fault tolerance and allows for parallel processing of events.</p><p class="ql-align-justify">5. Producers: These are client applications that publish events into topics. They can associate events with a key to ensure they go to the same partition. 6. Consumers: These are client applications that subscribe to topics and read events from them. They can read events as they occur or go back and read events from the beginning. To build an event streaming pipeline, we create topics, publish events using producers, and consume events using consumers. We can also use the Kafka command-line interface (CLI) to manage topics, producers, and consumers. Kafka provides a powerful and scalable solution for processing and analyzing streams of events.</p></div><div id="content-5"><h1 class="ql-align-justify"><strong>Installation</strong></h1><p class="ql-align-justify">In this lab we wil going to test the apache kafka using the docker compose. Here is the file of the docker compose, so you can test it immediately without thinking how to installed it.</p></div><div id="content-6"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">version: '3.8'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - "2181:2181"

  kafka:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1</code></pre></div><div id="content-7"><p>We can now running this docker compose by type in the terminal : docker-compose up -d. and then wait for all the setup to be completed. Once we completed we can now access the kafka container using this command:</p></div><div id="content-8"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">docker exec -it &lt;kafka-container-id&gt; bash</code></pre></div><div id="content-9"><p>Now we are inside of the container. Now we can create a topic inside this apache kafka. we simply create a topic named test. Here is the command in the terminal:</p></div><div id="content-10"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">kafka-topics --create --topic test --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1</code></pre></div><div id="content-11"><p>Once we create a topic now we can prepare other 2 terminal for the apache kafka demonstration. where the one terminal is used fot the producer and the other one is going to be a consumer. Lets begin with the first termnal and type:</p></div><div id="content-12"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">kafka-console-producer --topic test --bootstrap-server localhost:9092</code></pre></div><div id="content-14"><p>Now our producer topic is available in this terminal. we can type a few messages and hit&nbsp;<code>Enter</code>&nbsp;after each message. In the second terminal we are going to make the consumer. here is the command:</p></div><div id="content-15"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">docker exec -it &lt;kafka-container-id&gt; kafka-console-consumer --topic test --bootstrap-server localhost:9092 --from-beginning</code></pre></div><div id="content-20"><p>Now we should see the messages we produced in the previous step. Here is the output of what he have been doing so far</p></div><div id="content-21"><img style='width:720px;' src='https://firebasestorage.googleapis.com/v0/b/personal-blog-darmajr.appspot.com/o/blog-content%2Fkafka_1.png?alt=media&token=09d39473-77b7-46a3-ac9d-2dd44b252e1a'/></div><div id="content-22"><img style='width:720px;' src='https://firebasestorage.googleapis.com/v0/b/personal-blog-darmajr.appspot.com/o/blog-content%2Fkafka_2.png?alt=media&token=b3885d54-45ce-41a7-8b43-3eccc3c52241'/></div><div id="content-23"><h1>Conclusion</h1><p>We've successfully installed Apache Kafka using Docker and tested it by producing and consuming messages. This setup can be expanded with additional configurations and services as needed for our use case.</p></div>

In this lab we will be covering about what is the apache kafka. How it works and little bit cover about the installation and the basic workflow demonstration

Introduction to Apache Kafka

<div id="content-0"><h1><strong>Introduction</strong></h1><p>Previously we already install the apache kafka using the docker compose, create a simple workflow of how the apache is running. You can visit this site to learn the apache kafka installation and the basic workflow if you are not familir with these topic (<a href="https://barbarpotato.github.io/#/lab/d1c5319b-3019-4576-9368-d3757bf35c6a" target="_blank" style="color: inherit; background-color: transparent;">Introduction to Apache Kafka</a>). To create a Kafka Python client that interacts with your Kafka instance, you can use the&nbsp;<code style="color: rgb(248, 248, 242); background-color: rgb(39, 40, 34);">confluent-kafka</code>&nbsp;library, which is a high-performance Kafka client built on the librdkafka C library.</p><h3><strong>Install the Kafka Python Client</strong></h3><p>First, you need to install the&nbsp;<code style="color: rgb(248, 248, 242); background-color: rgb(39, 40, 34);">confluent-kafka</code>&nbsp;library. You can do this using pip. Note: You can create virtual environment of your project folder if you dont want to install this library to your global library package. by using the command: python -m venv &lt;the name of your virtual environment&gt;. then activating the venv using this command: source &lt;your_venv_name&gt;/Scripts/activate.</p></div><div id="content-1"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">pip install confluent-kafka</code></pre></div><div id="content-2"><h3><strong>Produce Message</strong></h3><p>Create a Python script to produce messages to the Kafka topic</p></div><div id="content-3"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">from confluent_kafka import Producer
import socket

def delivery_report(err, msg):
    if err is not None:
        print(f"Message delivery failed: {err}")
    else:
        print(f"Message delivered to {msg.topic()} [{msg.partition()}]")

p = Producer({'bootstrap.servers': 'localhost:9092', 'client.id': socket.gethostname()})

topic = 'test'

for i in range(10):
    p.produce(topic, key=str(i), value=f"Message {i}", callback=delivery_report)
    p.poll(0)

p.flush()</code></pre></div><div id="content-4"><p></p></div><div id="content-5"><h3><strong>Consume Messages</strong></h3><p>Create another Python script to consume messages from the Kafka topic.</p></div><div id="content-6"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">from confluent_kafka import Consumer, KafkaError

c = Consumer({
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'mygroup',
    'auto.offset.reset': 'earliest'
})

c.subscribe(['test'])

while True:
    msg = c.poll(1.0)

    if msg is None:
        continue
    if msg.error():
        if msg.error().code() == KafkaError._PARTITION_EOF:
            print(f'%% {msg.topic()} [{msg.partition()}] reached end at offset {msg.offset()}')
        elif msg.error():
            raise KafkaException(msg.error())
    else:
        print(f'Received message: {msg.value().decode("utf-8")}')

c.close()</code></pre></div><div id="content-7"><p>the consumer script initializes a Kafka consumer using the&nbsp;<code style="color: rgb(248, 248, 242); background-color: rgb(39, 40, 34);">Consumer</code>&nbsp;class, subscribes to the specified topic(s) with the&nbsp;<code style="color: rgb(248, 248, 242); background-color: rgb(39, 40, 34);">subscribe</code>&nbsp;method, and retrieves messages from the Kafka topic using the&nbsp;<code style="color: rgb(248, 248, 242); background-color: rgb(39, 40, 34);">poll</code>&nbsp;method. Finally, the&nbsp;<code style="color: rgb(248, 248, 242); background-color: rgb(39, 40, 34);">close</code>&nbsp;method closes the consumer and commits the final offsets. This setup allows you to produce and consume messages using your Python Kafka client. Now you can save the file.</p><p><br></p><p>To run the producer and consumer, open two terminal windows or tabs. In the first terminal, run the producer by executing&nbsp;<code style="color: rgb(248, 248, 242); background-color: rgb(39, 40, 34);">python producer.py</code>, and in the second terminal, run the consumer by executing&nbsp;<code style="color: rgb(248, 248, 242); background-color: rgb(39, 40, 34);">python consumer.py</code>. The producer script will send 10 messages to the&nbsp;<code style="color: rgb(248, 248, 242); background-color: rgb(39, 40, 34);">test</code>&nbsp;topic, while the consumer script will receive and print these messages. The producer script initializes a Kafka producer using the&nbsp;<code style="color: rgb(248, 248, 242); background-color: rgb(39, 40, 34);">Producer</code>&nbsp;class and sends messages to a specified topic with the&nbsp;<code style="color: rgb(248, 248, 242); background-color: rgb(39, 40, 34);">produce</code>&nbsp;method. A&nbsp;<code style="color: rgb(248, 248, 242); background-color: rgb(39, 40, 34);">delivery_report</code>&nbsp;callback function confirms message delivery, and the&nbsp;<code style="color: rgb(248, 248, 242); background-color: rgb(39, 40, 34);">poll</code>&nbsp;method serves delivery reports, which should be called regularly. The&nbsp;<code style="color: rgb(248, 248, 242); background-color: rgb(39, 40, 34);">flush</code>&nbsp;method waits for all messages in the producer queue to be delivered. On the other hand, the consumer script initializes a Kafka consumer using the&nbsp;<code style="color: rgb(248, 248, 242); background-color: rgb(39, 40, 34);">Consumer</code>&nbsp;class, subscribes to the specified topic(s) with the&nbsp;<code style="color: rgb(248, 248, 242); background-color: rgb(39, 40, 34);">subscribe</code>&nbsp;method, and retrieves messages from the Kafka topic using the&nbsp;<code style="color: rgb(248, 248, 242); background-color: rgb(39, 40, 34);">poll</code>&nbsp;method. Finally, the&nbsp;<code style="color: rgb(248, 248, 242); background-color: rgb(39, 40, 34);">close</code>&nbsp;method closes the consumer and commits the final offsets. This setup allows you to produce and consume messages using your Python Kafka client.</p><p><br></p><p>Here is the example output about what we are doing:</p></div><div id="content-8"><img style='width:720px;' src='https://firebasestorage.googleapis.com/v0/b/personal-blog-darmajr.appspot.com/o/blog-content%2Fkafka_python_client_1.png?alt=media&token=4c079070-9f99-436f-b81c-8b9d0dbada30'/></div><div id="content-9"><img style='width:720px;' src='https://firebasestorage.googleapis.com/v0/b/personal-blog-darmajr.appspot.com/o/blog-content%2Fkafka_python_client_2.png?alt=media&token=7a151ac1-e386-4aef-a99a-f2d72fe611fd'/></div><div id="content-10"><h1><strong>Conclusion</strong></h1><p>In conclusion, we successfully set up Apache Kafka using Docker, enabling a reliable environment for message streaming. Following this, we implemented a Kafka Python client using the&nbsp;<code style="color: rgb(248, 248, 242); background-color: rgb(39, 40, 34);">confluent-kafka</code>&nbsp;library, which provides high-performance Kafka producer and consumer capabilities. The producer script was designed to initialize a Kafka producer, send messages to a specified topic, and confirm message delivery through a callback function, while regularly polling for delivery reports and ensuring all messages are delivered with a flush operation. The consumer script was crafted to initialize a Kafka consumer, subscribe to the desired topic(s), retrieve messages, and close the consumer after committing final offsets. By running the producer and consumer scripts in separate terminal windows, you verified the setup by successfully sending and receiving messages. This end-to-end setup demonstrates a functional Kafka environment and a practical Python client for producing and consuming messages, laying a strong foundation for building more complex message-driven applications.</p></div>

In this lab we will bring back the apache kafka to work with the application environment. In this case we are going to use the apache kafka to a simple python application.

Build Kafka Python Client

<div id="content-1"><h1>What is GRPC?</h1><p>gRPC (Google Remote Procedure Call) is a high-performance, open-source framework developed by Google that enables efficient communication between distributed systems using remote procedure calls (RPCs). It leverages HTTP/2 for transport, Protocol Buffers (protobuf) for serialization, and supports features like multiplexing, streaming, and low-latency communication, making it ideal for modern microservices architectures and real-time applications. gRPC is increasingly used in this era due to its ability to handle complex, high-throughput workloads while ensuring consistent, language-agnostic communication between services, which is crucial for building scalable, resilient, and performant systems.</p><p>gRPC is highly relevant today due to its efficient communication using Protocol Buffers for serialization, which reduces message sizes and improves performance. It leverages HTTP/2 for lower latency and higher throughput, supports multiple programming languages, and provides strongly typed contracts for consistency. With built-in support for streaming, SSL/TLS for security, and automatic code generation, gRPC enhances development speed and ensures secure, real-time communication. Its compatibility with cloud-native platforms and microservices architectures further makes it a valuable tool for modern distributed systems.</p></div><div id="content-2"><img style='width:720px;' src='https://firebasestorage.googleapis.com/v0/b/personal-blog-darmajr.appspot.com/o/blog-content%2Fgrpc_architecture.png?alt=media&token=fff5674e-8a0b-4e46-805d-feed5b3cf0f5'/></div><div id="content-4"><p>The basic architecture of gRPC involves several key components that facilitate communication between a client and server. Here's a breakdown of the architecture:</p><h3>1.&nbsp;<strong>Protocol Buffers (Protobufs):</strong></h3><ul><li><strong>Definition Language</strong>: Protobufs are used to define the service methods and message types. These definitions are stored in&nbsp;<code>.proto</code>&nbsp;files.</li><li><strong>Serialization</strong>: Protobufs serialize data into a compact binary format, which is efficient for transmission.</li></ul><h3>2.&nbsp;<strong>Service Definition:</strong></h3><ul><li><strong>Service Interface</strong>: In the&nbsp;<code>.proto</code>&nbsp;file, services are defined with methods that can be called remotely by clients. Each method specifies the request and response message types.</li></ul><h3>3.&nbsp;<strong>Code Generation:</strong></h3><ul><li><strong>Client and Server Stubs</strong>: The&nbsp;<code>.proto</code>&nbsp;file is used to generate client and server code in various languages. These stubs handle the serialization and deserialization of messages and the underlying gRPC protocol details.</li></ul><h3>4.&nbsp;<strong>Client-Side:</strong></h3><ul><li><strong>Stub</strong>: The client uses a generated stub to invoke methods on the server. The stub provides a local representation of the remote service.</li><li><strong>Channel</strong>: The client creates a communication channel to the server, which manages the connection and handles the low-level details of the communication.</li></ul><h3>5.&nbsp;<strong>Server-Side:</strong></h3><ul><li><strong>Implementation</strong>: The server implements the service methods defined in the&nbsp;<code>.proto</code>&nbsp;file. These methods process incoming requests and generate responses.</li><li><strong>Server</strong>: The server listens for incoming requests, dispatches them to the appropriate service method implementations, and sends back responses.</li></ul><h3>6.&nbsp;<strong>Communication:</strong></h3><ul><li><strong>HTTP/2</strong>: gRPC uses HTTP/2 as the transport protocol, providing features like multiplexing, flow control, and header compression.</li><li><strong>Streams</strong>: gRPC supports different types of RPCs, including unary (single request/response), server streaming, client streaming, and bidirectional streaming.</li></ul></div><div id="content-5"><h1>Basic gRPC Workflow:</h1><ol><li><strong>Define the Service</strong>: Create a <code>.proto</code> file with service and message definitions.</li><li><strong>Generate Code</strong>: Use the <code>protoc</code> compiler to generate client and server code from the <code>.proto</code> file.</li><li><strong>Implement the Server</strong>: Write the server-side code to implement the service methods.</li><li><strong>Create the Client</strong>: Write the client-side code to call the service methods using the generated stub.</li><li><strong>Establish Communication</strong>: The client and server communicate over a channel using HTTP/2, with data serialized and deserialized using Protobufs.</li></ol></div><div id="content-8"><h3>In gRPC, the client can indeed send data to the server. gRPC supports several types of communication patterns, including:</h3><h3>1. Unary RPC:</h3><ul><li>The client sends a single request to the server and receives a single response.</li></ul></div><div id="content-9"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">rpc SayHello (HelloRequest) returns (HelloReply) {}</code></pre></div><div id="content-10"><h3>2. Server Streaming RPC:</h3><ul><li>The client sends a single request to the server and receives a stream of responses.</li></ul></div><div id="content-11"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">rpc ListFeatures (Rectangle) returns (stream Feature) {}</code></pre></div><div id="content-12"><h3>3. Client Streaming RPC:</h3><ul><li>The client sends a stream of requests to the server and receives a single response.</li></ul></div><div id="content-13"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">rpc RecordRoute (stream Point) returns (RouteSummary) {}</code></pre></div><div id="content-14"><h3>4. Bidirectional Streaming RPC:</h3><ul><li>Both the client and server send a stream of messages to each other.</li></ul></div><div id="content-15"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">rpc RouteChat (stream RouteNote) returns (stream RouteNote) {}</code></pre></div><div id="content-16"><h1>Example</h1></div><div id="content-19"><p>Let's try to do the key list that we define in above. in this example we are going to implement the unary RPC for the sake of the simplicity. and in this example we are going to implement the grpc within the node.js with the helper of our beloved database called MongoDb. for this example what we are going to achieve is the client side can search the book data from the mongodb database. To ensure that the client can get the data successfully, the client side need to get interact with our grpc server. in a nutshell, the server side grpc will serve the client side. To create a gRPC server in Node.js, we need to install the required packages:</p></div><div id="content-20"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">npm install @grpc/grpc-js @grpc/proto-loader
</code></pre></div><div id="content-21"><p>after the installation is complete, we then create a proto file. in this file we define the service and the message that we are going to build. here is the example code:</p></div><div id="content-22"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">syntax = "proto3";

service Book {
  rpc GetBook (GetBookRequest) returns (BookReply) {}
}

message GetBookRequest {
  string Title = 1;
}

message BookReply {
  string Title = 1;
  string Author = 2;
  int32 Published = 3;
  string Language = 4;
  string Id = 5;
  int32 Sales = 6;
}</code></pre></div><div id="content-23"><p>the proto file is ready. now we can load it up trough our core script to running our node server. dont forget that we are using the mongodb database. so the first step is to setup the connection between the mongodb and the server, and then we build up the grpc service in a top of mongodb service.</p></div><div id="content-32"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">const grpc = require('@grpc/grpc-js');
const protoLoader = require('@grpc/proto-loader');
const path = require('path');
const { MongoClient } = require('mongodb');

// Load the protobuf file
const PROTO_PATH = path.resolve(__dirname, 'service.proto');
const packageDefinition = protoLoader.loadSync(PROTO_PATH);
const proto = grpc.loadPackageDefinition(packageDefinition);

// MongoDB connection URL and Database name
const url = 'mongodb://localhost:27017';
const dbName = 'Book';

let db;
let client;

// Connect to MongoDB and start gRPC server
async function main() {
    const client = new MongoClient(url);

    try {
        await client.connect();
        console.log('Connected to MongoDB');
        db = client.db(dbName);

        // Start gRPC server after MongoDB connection
        startGrpcServer();
    } catch (e) {
        console.error('Failed to connect to MongoDB', e);
    } finally {
        // Ensure MongoClient is closed if needed
        // Uncomment if you need to close the client here:
        // await client.close();
    }
}


// gRPC service method
function GetBook(call, callback) {
    console.log('Received request:', call.request);
    const { Title } = call.request;
    const collection = db.collection('list');

    console.log('Performing findOne query with Book:', Title);
    collection.findOne({ Book: Title }, { maxTimeMS: 5000 })
        .then(result =&gt; {
            console.log('findOne query completed');
            if (result) {
                const response = {
                    Title: result.Book,
                    Author: result['Author(s)'], // Adjusted field names
                    Published: result['First published'],
                    Language: result['Original language'],
                    Id: result._id.toString(),
                    Sales: result['Approximate sales in millions']
                };
                console.log('Sending response:', response);
                callback(null, response);
            } else {
                console.log('Book not found');
                callback(null, { Title: "Book not found" });
            }
        })
        .catch(err =&gt; {
            console.error('Error fetching data:', err);
            callback(err, null);
        });
}


// Start the gRPC server
function startGrpcServer() {
    const server = new grpc.Server();
    server.addService(proto.Book.service, { GetBook: GetBook });
    const port = '50051';
    server.bindAsync(`0.0.0.0:${port}`, grpc.ServerCredentials.createInsecure(), (error, port) =&gt; {
        if (error) {
            console.error(`Failed to bind server: ${error.message}`);
            process.exit(1); // Exit process if server fails to start
        }
        console.log(`Server running at http://0.0.0.0:${port}`);
    });
}

// Run the main function
main();</code></pre></div><div id="content-33"><p>We can now run the server. by using the command node &lt;the name of your server code file&gt;</p></div><div id="content-34"><img style='width:720px;' src='https://firebasestorage.googleapis.com/v0/b/personal-blog-darmajr.appspot.com/o/blog-content%2Fgrpc-step-1.png?alt=media&token=cef277a6-d5e2-45d9-80f2-6871e3715089'/></div><div id="content-35"><p>The setup and implementation of the server.js is completed. now we move to the code of the client side. which is more simple. here is the code to call the available service from the server side that we implement previously. where in here we can access the GetBook service. and try to search the field named book with value&nbsp;Where the Crawdads Sing</p></div><div id="content-36"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">const grpc = require('@grpc/grpc-js');
const protoLoader = require('@grpc/proto-loader');
const path = require('path');

// Load the protobuf file
const PROTO_PATH = path.resolve(__dirname, 'service.proto');
const packageDefinition = protoLoader.loadSync(PROTO_PATH);
const proto = grpc.loadPackageDefinition(packageDefinition);

// Create a new gRPC client
const client = new proto.Book('localhost:50051', grpc.credentials.createInsecure());

// Call the service
client.GetBook({ Title: 'Where the Crawdads Sing' }, (error, response) =&gt; {
    if (error) {
        console.error(`Error: ${error.message}`);
    } else {
        console.log(`Response: ${JSON.stringify(response)}`);
    }
});
</code></pre></div><div id="content-37"><p>We can run this client side. and it the result of the book that we search is delivered to the client side.</p></div><div id="content-38"><img style='width:720px;' src='https://firebasestorage.googleapis.com/v0/b/personal-blog-darmajr.appspot.com/o/blog-content%2Fgrpc-step-2.png?alt=media&token=85d657cc-6260-492d-a70e-62d26b038da2'/></div><div id="content-40"><h1>Conclusion</h1><p>gRPC is a powerful framework for building efficient, high-performance communication between distributed systems. By leveraging HTTP/2 and Protocol Buffers (protobuf), gRPC enables low-latency, scalable interactions ideal for microservices and real-time applications. In this example, we demonstrated how to set up a gRPC server in Node.js that interacts with a MongoDB database. We defined a service using Protocol Buffers, implemented the server to handle requests, and created a client to query book data. This approach allows for efficient data retrieval and robust communication between client and server, showcasing gRPC's capabilities in modern software development. By understanding gRPC's architecture and workflow, and implementing it in a real-world scenario, you can build scalable, high-performance systems that handle complex workloads with ease.</p></div>

We will explore about the grpc, which is the better version of the basic HTTP1.1 and make communications more faster between client-server

Create GRPC With MongoDB & Node.js

<div id="content-0"><h1><strong>Introduction</strong></h1><p>In the fast-paced world of software development, delivering high-quality code consistently is crucial. One of the methodologies that can help achieve this goal is Test-Driven Development (TDD). TDD is not just a testing technique; it’s a design methodology that can profoundly impact how you write and maintain code. This blog post will explore what TDD is, its benefits, and how you can start incorporating it into your development workflow.</p></div><div id="content-2"><h1><strong>What is Test-Driven Development (TDD)?</strong></h1><p>Test-Driven Development is a software development process where you write tests before writing the actual code. The cycle typically follows three steps, known as the <strong>Red-Green-Refactor</strong> cycle:</p><ol><li><strong>Red</strong>: Write a test for a new functionality. Since the functionality is not yet implemented, the test will fail (hence the "red" phase).</li><li><strong>Green</strong>: Write the minimum amount of code required to pass the test. At this stage, you aim to make the test pass, even if the solution isn’t perfect.</li><li><strong>Refactor</strong>: Improve the code without changing its functionality. This might involve cleaning up code, optimizing performance, or improving readability.</li></ol></div><div id="content-3"><p>This cycle is repeated for every piece of functionality you want to add to your application.</p></div><div id="content-4"><h1><strong>Benefits of TDD</strong></h1><p><strong>Better Code Quality</strong>: Test-Driven Development (TDD) promotes writing only the code needed to pass predefined tests, which helps prevent overengineering. This focus on minimalism means developers are less likely to introduce unnecessary complexity. Moreover, the tests themselves act as a safety net, identifying bugs early in the development process. By catching issues before they escalate, TDD ensures that the code remains clean, efficient, and easy to understand.</p><p><strong>Improved Design</strong>: TDD forces developers to think about the design and functionality of their code before they start writing it. This upfront consideration leads to better-structured, more modular, and decoupled code. Since the tests are written first, they guide the design, ensuring that each piece of the codebase is independently testable and reusable. This modularity not only makes the code easier to maintain but also simplifies future enhancements and refactoring.</p><p><strong>Faster Debugging</strong>: One of the standout benefits of TDD is its ability to streamline the debugging process. When a test fails, it provides an immediate indication of where the problem lies, allowing developers to pinpoint the issue quickly. Instead of spending hours sifting through code to find the root cause of a bug, developers can focus on the specific area that triggered the test failure, significantly reducing debugging time.</p><p><strong>Documentation</strong>: TDD naturally results in a comprehensive suite of tests that serve as living documentation for the codebase. These tests provide concrete examples of how functions and classes are expected to behave, making it easier for other developers (or even your future self) to understand the code’s purpose and usage. Unlike traditional documentation, which can become outdated, these tests are constantly updated as the code evolves, ensuring they remain relevant and accurate.</p><p><strong>Increased Confidence</strong>: With a robust suite of tests in place, developers can refactor or extend their code with greater confidence. The tests act as a safeguard, ensuring that any unintended side effects are caught immediately. This level of assurance is particularly valuable when making significant changes to the codebase, as it minimizes the risk of introducing new bugs and helps maintain the overall stability of the application.</p></div><div id="content-5"><h1><strong>Common Misconceptions About TDD</strong></h1><p><strong>TDD is just about testing</strong>: While the name might suggest that TDD is primarily focused on testing, it’s actually more about guiding the design of your code. The tests you write before the implementation help shape the structure and functionality of the code. TDD drives the development process, ensuring that the code is built to meet the specific requirements laid out in the tests, which results in a more thoughtful and deliberate design.</p><p><strong>TDD slows down development</strong>: At first glance, TDD may seem to slow down development, as it requires writing tests before the actual code. However, this initial investment pays off in the long run. By catching bugs early and reducing the time spent on debugging, TDD often leads to faster overall development. Additionally, the improved code quality and design that result from TDD can reduce the need for extensive refactoring later in the project, further accelerating the development process.</p><p><strong>TDD is only for large projects</strong>: Another common misconception is that TDD is only beneficial for large, complex projects. In reality, TDD can be applied to projects of any size. Even in small projects, TDD helps ensure that the codebase remains maintainable and free of bugs. By adopting TDD from the start, developers can build a strong foundation that makes it easier to scale the project as it grows. The principles of TDD—writing clean, testable code—are universally applicable, regardless of the project's scope.</p></div><div id="content-7"><p>Here's a simple example of a Test-Driven Development (TDD) workflow using <code>unittest</code> in Python. This example demonstrates the TDD cycle: <strong>Red-Green-Refactor</strong>.</p></div><div id="content-8"><h1><strong>Scenario</strong>:</h1><p>You want to create a function that returns the factorial of a number.</p></div><div id="content-9"><h3><strong>Step 1: Write a Failing Test (Red)</strong></h3><p>First, you write a test for the functionality you want to implement, even though the function doesn’t exist yet. This test will naturally fail because the function isn’t defined.</p></div><div id="content-10"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">import unittest

class TestFactorialFunction(unittest.TestCase):
    def test_factorial_of_5(self):
        result = factorial(5)
        self.assertEqual(result, 120)

if __name__ == '__main__':
    unittest.main()</code></pre></div><div id="content-11"><p><strong>Explanation</strong>:</p><ul><li>This test checks if the <code>factorial</code> function correctly returns 120 when called with the argument <code>5</code>.</li><li>Since the <code>factorial</code> function doesn't exist, running this test will produce an error.</li></ul></div><div id="content-12"><h3><strong>Step 2: Implement the Minimum Code to Pass the Test (Green)</strong></h3><p>Next, you implement the <code>factorial</code> function, just enough to pass the test.</p></div><div id="content-13"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">def factorial(n):
    if n == 0:
        return 1
    else:
        return n * factorial(n - 1)
</code></pre></div><div id="content-14"><p>Now, if you run the test, it should pass:</p></div><div id="content-15"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">$ python test_factorial.py</code></pre></div><div id="content-16"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">.
----------------------------------------------------------------------
Ran 1 test in 0.001s

OK
</code></pre></div><div id="content-17"><h3><strong>Step 3: Refactor the Code</strong></h3><p>Finally, you can refactor the code if necessary to improve its design or efficiency. In this case, the code is already quite clean, so no major refactoring is needed. However, if there were redundant code or opportunities to optimize, you would do that in this step.</p><p>You could also add more tests to cover additional cases, such as the factorial of <code>0</code> or negative numbers, ensuring your function is robust:</p></div><div id="content-18"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">class TestFactorialFunction(unittest.TestCase):
    def test_factorial_of_5(self):
        result = factorial(5)
        self.assertEqual(result, 120)

    def test_factorial_of_0(self):
        result = factorial(0)
        self.assertEqual(result, 1)

    def test_factorial_of_1(self):
        result = factorial(1)
        self.assertEqual(result, 1)

    def test_factorial_of_negative(self):
        with self.assertRaises(ValueError):
            factorial(-1)

if __name__ == '__main__':
    unittest.main()
</code></pre></div><div id="content-19"><p><strong>Explanation</strong>:</p><ul><li>These additional tests handle edge cases, such as when <code>n</code> is <code>0</code>, <code>1</code>, or negative.</li><li>The <code>test_factorial_of_negative</code> test checks that the function raises a <code>ValueError</code> when given a negative input.</li></ul><p>If the function doesn't handle these cases yet, you would modify it:</p></div><div id="content-20"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">def factorial(n):
    if n &lt; 0:
        raise ValueError("Factorial is not defined for negative numbers")
    elif n == 0 or n == 1:
        return 1
    else:
        return n * factorial(n - 1)</code></pre></div><div id="content-21"><h1>Conclusion</h1><p>From this simple scenario, we know that the Test-Driven Development is more than just a testing technique—it’s a powerful design methodology that can lead to cleaner, more maintainable, and bug-free code. By adopting TDD, you can improve your development process, catch bugs early, and build software that stands the test of time. If you haven’t tried TDD yet, now is the perfect time to start. Happy coding!</p></div>

In the fast-paced world of software development, delivering high-quality code consistently is crucial. One of the methodologies that can help achieve this goal is Test-Driven Development (TDD). 

Test-Driven Development with Python

<div id="content-0"><h1>What is Redis?</h1><p>Redis is a key-value store that operates entirely in memory, making it significantly faster compared to traditional databases that store data on disk. Redis is known as a <strong>NoSQL</strong> database and is often used in applications that require high-speed data access.</p></div><div id="content-1"><h2>Advantages of Redis</h2><p>Redis offers several advantages that make it an appealing choice for many developers:</p><h3>1. <strong>Speed</strong></h3><p>Redis stores data in memory rather than on disk. This means read and write operations occur in milliseconds, making Redis ideal for applications that require very fast response times.</p><h3>2. <strong>Support for Various Data Structures</strong></h3><p>Redis doesn’t just store strings but also supports various data structures such as hashes, lists, sets, and sorted sets. This provides flexibility in how data can be organized and accessed.</p><h3>3. <strong>Data Persistence</strong></h3><p>Although Redis is an in-memory database, it has options to save data to disk, either periodically or based on direct commands. This ensures that data is not lost when Redis is shut down or crashes.</p><h3>4. <strong>Replication</strong></h3><p>Redis supports master-slave replication, where data can be replicated to multiple slave servers. This not only increases data availability but also allows Redis to scale the read load.</p><h3>5. <strong>Pub/Sub Mechanism</strong></h3><p>Redis features a publish/subscribe mechanism that allows systems to communicate in real-time by subscribing to certain channels and receiving messages whenever they are published to those channels. This is particularly useful for building real-time applications such as chat systems and notifications.</p></div><div id="content-2"><h1>Using Redis in Applications</h1><p>Redis is often used in various scenarios, some of which include:</p><h3>1. <strong>Cache</strong></h3><p>Redis is frequently used as a cache to store database query results or frequently accessed data, reducing the load on the main database and improving application response times.</p><h3>2. <strong>Session Store</strong></h3><p>Many web applications use Redis to store user sessions. Using Redis allows sessions to be quickly accessed and deleted when no longer needed.</p><h3>3. <strong>Queue Management</strong></h3><p>Redis is used to manage job queues in distributed systems, such as background jobs, task scheduling, and more.</p><h3>4. <strong>Real-time Analytics</strong></h3><p>Due to its speed, Redis is often used for real-time data processing, such as counting website visitors or monitoring application performance.</p></div><div id="content-5"><h1>Installation</h1><p>In this lab, we are going to use the Redis services provided by vercel platform. To integrate Redis from Vercel into your Next.js project, you can follow these steps:</p><ol><li>Go to the Vercel Redis dashboard and create a new Redis instance.</li><li>Configure the Redis instance with the desired settings.</li><li>After creating the Redis instance, you will receive a connection URL, which typically looks like <code>rediss://:&lt;password&gt;@&lt;host&gt;:&lt;port&gt;</code>.</li></ol></div><div id="content-6"><h3>Install Redis Client for Next.js</h3><p>You will need a Redis client to connect to the Redis instance in your Next.js project. You can use the <code>ioredis</code> library, a popular Redis client for Node.js.</p></div><div id="content-7"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">npm install ioredis</code></pre></div><div id="content-8"><h3>Configure Environment Variables</h3><p>Add the Redis connection URL to your environment variables. Create a <code>.env.local</code> file in the root of your Next.js project:</p></div><div id="content-9"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">REDIS_URL=rediss://:&lt;password&gt;@&lt;host&gt;:&lt;port&gt;
</code></pre></div><div id="content-10"><h3>Connect to Redis in Next.js</h3><p>Create a utility function to handle Redis connections. You can create a file like <code>redis.js</code> inside the <code>lib</code> folder (or any other suitable location in your project):</p></div><div id="content-11"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">// lib/redis.js
import Redis from 'ioredis';

let client;

if (!client) {
  client = new Redis(process.env.REDIS_URL);
}

export default client;</code></pre></div><div id="content-12"><h3>Use Redis in API Routes or Server Components</h3><p>You can now use this Redis client in your API routes or server components. Here is an example of an API route using Redis:</p></div><div id="content-13"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">import redis from "@/app/libs/redis";

const getHandler = async (req, res) =&gt; {

    try {

        // Get the value from Redis
        const value = await redis.get("Darwin AI: Active");

        // Send the response of a darwin service status
        if (value) {
            res.status(200).json({ "Darwin": "Active" });
        } else {
            res.status(200).json({ "Darwin": "Inactive" });
        }
    } catch (error) {
        res.status(500).json({ error });
    }
}

const postHandler = async (req, res) =&gt; {

    try {
        const current_timestamp = Date.now();

        // Set a value in Redis. make it expired after 10 minutes
        await redis.set("Darwin AI: Active", current_timestamp, 'EX', 600);

        // Get the value from Redis
        const value = await redis.get("Darwin AI: Active");

        // Send the response of a darwin service status
        if (value) {
            res.status(200).json({ "Darwin": "Active" });
        } else {
            res.status(200).json({ "Darwin": "Inactive" });
        }
    } catch (error) {
        res.status(500).json({ error });
    }
}


const handler = async (req, res) =&gt; {
    if (req.method === 'GET') {
        return getHandler(req, res);
    }
    else if (req.method === 'POST') {
        return postHandler(req, res);
    }
    else if (req.method === 'DELETE') {
        return deleteHandler(req, res);
    }
    else {
        return res.status(405).end(`Method ${req.method} Not Allowed`);
    }
};

export default handler;</code></pre></div><div id="content-14"><h1>Conclusion</h1><p>Redis is a powerful and versatile tool in the world of application development. With its high speed, support for various data structures, and advanced features like persistence and replication, Redis can be used to significantly enhance application performance. In today’s fast-paced world, Redis offers an efficient solution to many challenges faced by developers.</p></div>

Redis, which stands for Remote Dictionary Server, is an in-memory data structure store that is commonly used as a database, cache, and message broker. 

Build Redis in Next.js

<div id="content-0"><h1>Understanding k6: A Load and Performance Testing Tool</h1><p>When developing an application, ensuring it can handle high traffic is crucial. This is where load testing and performance testing come into play, and <strong>k6</strong> is a powerful tool designed for this purpose. By using k6, you can assess whether your application can handle heavy traffic and perform optimally under various conditions.</p></div><div id="content-1"><h1>What is k6?</h1><p>k6 is a command-line based tool specifically designed for load testing and performance testing. It allows you to simulate multiple users accessing your application, which helps identify potential issues when traffic spikes.</p><h2><strong>Key Points About k6:</strong></h2><ul><li><strong>Terminal-Based:</strong> k6 operates directly through the terminal.</li><li><strong>JavaScript Testing:</strong> Tests are written in JavaScript, making it accessible for web developers.</li></ul><h2>What k6 Does Not Do</h2><p>While k6 is a robust tool, it has its limitations:</p><ul><li><strong>No Browser Interaction:</strong> k6 does not run through a browser.</li><li><strong>No Node.js Dependency:</strong> Unlike many JavaScript tools, k6 doesn't rely on Node.js but is built using Golang.</li><li><strong>No npm Modules Support:</strong> If you require additional modules, you can't install them via <code>npm install</code>. Instead, you must include the files directly in your project.</li></ul><h2>Limitations of k6</h2><p>k6 uses Golang's Goja library to execute JavaScript, which means it doesn’t support some of the features available in Node.js. This limitation requires a different approach when setting up your testing environment.</p></div><div id="content-2"><h2>Installation</h2><p>You can easily install k6 by following the instructions on their <a href="https://k6.io/docs/get-started/installation/" target="_blank">official installation page</a>.</p></div><div id="content-3"><h1>Initial Project Setup</h1><p>To get started with k6, you can set up a Node.js project:</p><ul><li>Initialize the project with <code>npm init</code> and set the type to <code>module</code>.</li><li>Install k6 as a module in your project.</li></ul><h2>k6 Library</h2><p>k6 comes with its own library that simplifies the process of creating tests. However, remember that the npm package for k6 doesn't contain actual JavaScript code—it’s mostly metadata. The k6 tool itself is built on Golang, so it doesn’t run on Node.js.</p></div><div id="content-4"><p>If you inspect the k6 module, you’ll find that it’s essentially an empty package with just a <code>package.json</code> format.</p><h3>Writing k6 Scripts</h3><p>Scripts in k6 are written in JavaScript and contain the code for the performance test along with the test scenarios you wish to execute. You can create a script manually or use a command to generate a basic script:</p></div><div id="content-5"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">k6 new src/ping.js
</code></pre></div><div id="content-6"><p>This command will create a new script at the specified location with a basic setup for performance testing.	</p></div><div id="content-7"><h3>Running a Script</h3><p>To execute the performance test, use the following command:</p></div><div id="content-8"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">k6 run lokasi/file.js
</code></pre></div><div id="content-10"><h1>Checking the code</h1></div><div id="content-11"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">import http from 'k6/http';
import { sleep } from 'k6';

export const options = {
  stages: [
    { duration: '10s', target: 10 },
    { duration: '20s', target: 30 },
    { duration: '30s', target: 50 },
    { duration: '10s', target: 0 }
  ]
  // // A number specifying the number of VUs to run concurrently.
  // vus: 10,
  // // A string specifying the total duration of the test run.
  // duration: '30s',

  const body = {
    name: "Darmawan"
  };
};

export default function () {

  http.post("https://httpbin.test.k6.io/post", JSON.stringify(body), {
    headers: {
      "Content-Type": "application/json",
      "Accept": "application/json"
    }
  })

  http.get('https://cerberry-backend.vercel.app/blogs/all');
  sleep(0.1);
}
</code></pre></div><div id="content-12"><p>we'll explore a simple yet effective k6 script designed to conduct load testing on an API. The script simulates multiple virtual users (VUs) interacting with the application, allowing us to observe how it performs under varying levels of traffic. To start, the script imports the necessary modules from k6—<code>http</code> for making HTTP requests and <code>sleep</code> to simulate user think time by pausing execution for a short duration.</p><p>The core of the script lies in the <code>options</code> object, where we define a series of stages that dictate how the load should be applied. The test begins by gradually ramping up to 10 virtual users over the first 10 seconds. It then continues to increase to 30 users over the next 20 seconds, and finally reaches 50 users over a 30-second period. After this peak, the number of users is gradually reduced back to zero over the last 10 seconds. This staged approach provides valuable insights into the scalability and robustness of the application as it experiences increasing and then decreasing traffic.</p><p>Within the <code>default</code> function, the script defines two key actions that each virtual user will perform. First, a POST request is made to <code>https://httpbin.test.k6.io/post</code>, sending a JSON payload with a name value of "Darmawan". This simulates a scenario where a user might submit data to the server, such as filling out a form. Immediately after, a GET request is sent to <code>https://cerberry-backend.vercel.app/blogs/all</code>, retrieving a list of blogs. This sequence mirrors typical user behavior in a web application, where data submission is often followed by data retrieval or refreshing content. To simulate a brief delay between these actions, the script includes a <code>sleep(0.1)</code> function, representing a short pause of 0.1 seconds.</p><p>To execute this script, you would simply run it using the k6 command in your terminal. This approach allows you to assess how well your application can handle varying levels of traffic, identify potential bottlenecks, and ensure that it remains performant under load. By gradually increasing and then decreasing the load, this script provides a comprehensive view of your application's performance, helping you make informed decisions about optimizations and improvements.</p></div><div id="content-13"><h2>Understanding the Summary Output</h2><p>After running a performance test, k6 generates a summary output that details the test results. By default, this output is displayed in the terminal, but you can also export it as a JSON file:</p></div><div id="content-14"><pre style="background-color: black; color: white; padding:10px; border-radius: 5px;"><code style="color: white;">k6 run file/script.js --summary-export output.json
</code></pre></div><div id="content-15"><p>You can find more information on the output metrics in the <a href="https://grafana.com/docs/k6/latest/using-k6/metrics/reference/" target="_blank">Grafana documentation</a>.</p></div><div id="content-16"><h2>Using the Web Dashboard</h2><p>k6 also offers a Web Dashboard for real-time monitoring of your tests. To enable this feature, you need to activate it through an environment variable. Detailed instructions are available in the <a href="https://grafana.com/docs/k6/latest/results-output/web-dashboard/" target="_blank">Web Dashboard documentation</a>.</p><h2>Using Stages for Dynamic Load Testing</h2><p>k6 provides a powerful feature called <strong>Stages</strong>, which allows you to increase or decrease the number of virtual users during the test. This is particularly useful for simulating different traffic patterns over time. You can learn more about this feature in the <a href="https://grafana.com/docs/k6/latest/using-k6/k6-options/reference/#stages" target="_blank">Stages documentation</a>.</p><p>By using k6 effectively, you can ensure your application is well-prepared to handle real-world traffic, preventing potential issues before they reach your users.</p></div>

k6 is a command-line based tool specifically designed for load testing and performance testing. It allows you to simulate multiple users accessing your application, which helps identify potential issues when traffic spikes.

🚀Darmawan

Home

What Are You Looking For?