Technology-driven Software Engineer with 2.5+ years of experience in translating business requirements and functional specifications into code modules and software solutions with a strong focus on high-quality design, and performance.

**doodooti** - Your new favorite spot **!**

Featured Posts


<div style={{ marginTop: '20px' }}>
  <h3>Introduction</h3>
  <div style={{ padding: '5px' }}>
    <p>
      When building data-driven applications in Node.js, we often rely on ORMs
      or query builders to simplify database operations. However, there’s one
      silent performance killer that can sneak into even the most polished
      systems the N + 1 query problem.
    </p>
  </div>
</div>

<div style={{ marginTop: '20px' }}>
  <h3>What Is the N + 1 Problem</h3>
  <div style={{ padding: '5px' }}>
    <p>
      The N + 1 query problem occurs when the application executes one query to
      fetch a list of records (N) and then performs one additional query for
      each of those records (+1) to load related data.
    </p>
    <p>
      It’s a pattern that seems harmless in development but becomes a nightmare
      in production as your data scales.
    </p>

<p>
  Consider a scenario where we want to display users alongside their posts. If
  we use the <b>the naive way</b> by fetching all users first and then
  separately fetch posts for each user, we will end up to the N+1 problem
</p>

```javascript
// Step 1: Fetch all users
const users = await db.query('SELECT * FROM users'); // 1 query

// Step 2: For each user, fetch their posts
for (const user of users) {
  const posts = await db.query('SELECT * FROM posts WHERE user_id = ?', [
    user.id,
  ]); // N queries
  user.posts = posts;
}
```

<p>
  If we have <strong>100 users</strong>, this approach executes
  <strong>101 queries</strong> one to fetch users and one per user to get their posts.
</p>

<p>
  This is might not be visible in small datasets, but as our data grows the
  performance collapses because:
</p>

<ol>
  <li>
    Each query adds <strong>network latency</strong>
  </li>
  <li>
    The database handles <strong>repeated work</strong>
  </li>
  <li>
    Waste <strong>connections</strong> and <strong>CPU cycles</strong>
  </li>
  <li>
    The app becomes <strong>slow under load</strong> (especially in{' '}
    <strong>APIs</strong> or <strong>GraphQL resolvers</strong>)
  </li>
</ol>
</div>
</div>

<div style={{ marginTop: '20px' }}>
  <h3>The Solution</h3>
  <div style={{ padding: '5px' }}>
 <p>The most efficient way to solve the N + 1 problem is to use the
      <strong>eager loading</strong> that is to fetch related data in one query via
      relations and JOINs instead of one per record, in <strong>Sequelize </strong>
       that looks like this: </p>

```javascript
const users = await User.findAll({
  include: [{ model: Post }],
});
```

<p>This generate a join query:</p>

```sql
SELECT users.*, posts.*
FROM users
LEFT JOIN posts ON users.id = posts.user_id;
```

<p>
  With <strong>TypeORM</strong> we can easily fall into the same trap but we can
  solve it with <strong>eager loading</strong>:
</p>

```javascript
const users = await userRepository.find({
  relations: ['posts'],
});
```

<p>
  If the eager loading technique is not an option, there are other Alternative
  strategies to reduce query overhead
</p>

<h4 style={{ marginTop: '20px' }}>Batching:</h4>

<p>
  Group multiple individual queries into a single, optimized query by using{' '}
  <b>IN</b> clauses to fetch all related records at once, rather than executing
  separate queries for each parent entity
</p>

```javascript
const users = await db.query('SELECT * FROM users');
const userIds = users.map((u) => u.id);

const posts = await db.query('SELECT * FROM posts WHERE user_id IN (?)', [
  userIds,
]);
```

<p>
  Then manually group posts by user in the code, this approach is great when we
  can’t use ORM relations
</p>

<h4 style={{ marginTop: '20px' }}>Caching:</h4>

<p>
  While this approach doesn’t remove the N + 1 pattern, but it mitigates the
  performance cost by using an in-memory cache (like Redis) to store frequently
  fetched related data
</p>

```javascript
const cachedPosts = await redis.get(`posts:${userId}`);
```

<h4 style={{ marginTop: '20px' }}>Data Loader Pattern (for GraphQL APIs):</h4>

<p>
  In the case of using <b>GraphQL</b> the <b>DataLoader</b> library from
  <b>Facebook</b> is designed to batch and cache requests automatically.
</p>

<p style={{ marginTop: '5px' }}>Without DataLoader (N+1 Problem):</p>

```javascript
const resolvers = {
  Query: {
    users: async () => {
      // 1 query to get all users
      return await db.query('SELECT * FROM users');
    },
  },
  User: {
    posts: async (user) => {
      // This runs for EACH user! (N queries)
      return await db.query('SELECT * FROM posts WHERE user_id = ?', [user.id]);
    },
  },
};
```

<p>
  To integrate DataLoader into our GraphQL resolvers and take advantage of its
  automatic request batching, we need to establish the proper configuration
</p>

```javascript
const DataLoader = require('dataloader');

// Batch function: receives array of user IDs
const batchGetPosts = async (userIds) => {
  console.log('Batching user IDs:', userIds); // [1, 2, 3, ...]

  // Single query with IN clause
  const posts = await db.query('SELECT * FROM posts WHERE user_id IN (?)', [
    userIds,
  ]);

  // Group posts by user_id
  const postsByUserId = userIds.map((userId) =>
    posts.filter((post) => post.user_id === userId)
  );

  return postsByUserId;
};

// Create DataLoader (typically done per-request)
const createLoaders = () => ({
  postsLoader: new DataLoader(batchGetPosts),
});
```

<p>Then use the DataLoader in the Resolvers:</p>

```javascript
const resolvers = {
  Query: {
    users: async () => {
      // 1 query to get all users
      return await db.query('SELECT * FROM users');
    },
  },
  User: {
    posts: async (user, args, context) => {
      // Instead of querying directly, use the loader
      // DataLoader batches all these calls automatically
      return await context.loaders.postsLoader.load(user.id);
    },
  },
};
```

<p>The final step is to add the DataLoader to the Context</p>

```javascript
const { ApolloServer } = require('apollo-server');

const server = new ApolloServer({
  typeDefs,
  resolvers,
  context: ({ req }) => ({
    // This function runs ONCE per request
    // Everything returned here is available in all resolvers
    user: req.user, // Current authenticated user
    db: databaseConnection, // Database connection
    loaders: createLoaders(), // DataLoaders
    token: req.headers.authorization,
  }),
});
```

<p>
  With this configuration in place the N+1 problem is solved. Every request
  creates fresh loaders so no data bleeds between requests. Multiple resolvers
  can call the same loader and DataLoader automatically batches those calls into
  a single efficient database query
</p>

<aside
  style={{
    borderLeft: '4px solid #2b6cb0',
    padding: '12px',
    background: '#f1f5f9',
    width: '100%',
  }}
>
  <strong>Note:</strong>
  <p style={{ margin: '8px 0 0' }}>
    The Context is an object that's shared across all resolvers during a single
    GraphQL request
  </p>
</aside>

</div>
</div>

<div style={{ marginTop: '20px' }}>
  <h3>Detecting the N + 1 Problem</h3>
  <div style={{ padding: '5px' }}>
    <p>
      The N+1 problem loves to hide during development with small datasets but
      becomes a nightmare in production. Here's how to catch it before it causes
      trouble:
    </p>
    <ol>
      <li>
        <strong>Enable query logging:</strong> Turn on SQL logging in the ORM (
        <strong>Sequelize</strong>, <strong>Prisma</strong>,{' '}
        <strong>TypeORM</strong>) and watch the console. When the same query
        runs over and over with just the ID changing that's the N+1 problem
        right there. Something like{' '}
        <code>SELECT * FROM posts WHERE user_id = 1</code> then{' '}
        <code>user_id = 2</code> then <code>user_id = 3</code> is a dead
        giveaway.
      </li>
      <li>
        <strong>Use performance monitoring tools:</strong> Tools like{' '}
        <strong>New Relic</strong>, <strong>Datadog</strong> or{' '}
        <strong>AppSignal</strong> show exactly how many database queries each
        request makes. They highlight endpoints that are hammering the database
        and make it super obvious when something's wrong. These are especially
        useful because they catch problems in production with real data volumes.
      </li>
      <li>
        <strong>Check database metrics:</strong> The database can tell a lot
        about what's happening. PostgreSQL has{' '}
        <strong>pg_stat_statements</strong> and MySQL has query logs that track
        which queries run most frequently. Sometimes the database side catches
        things the application side misses.
      </li>
    </ol>
  </div>
</div>

<div style={{ marginTop: '20px' }}>
  <h3>Conclusion</h3>
  <div style={{ padding: '5px' }}>
    <p>
      The N + 1 problem is one of those issues that doesn’t show up in
      development until it does and suddenly our API is taking seconds to
      respond but by learning how to recognize and prevent it early we will
      build faster, more efficient and scalable Apps
    </p>
  </div>
</div>


The N + 1 problem in Node.js


<div style={{ marginTop: '20px' }}>
  <h3>Introduction</h3>
  <div style={{ padding: '5px' }}>
    Amazon EventBridge Scheduler is a serverless tool that helps us automate
    tasks across AWS. We can schedule millions of tasks from one place,
    triggering any AWS service we need like running Lambda functions, starting
    EC2 instances, or processing data at specific times. It's fully managed and
    scales automatically, so we just focus on what we want to accomplish and
    when, without worrying about maintaining infrastructure.
  </div>
</div>

<div style={{ marginTop: '20px' }}>
  <h3>Why EventBridge Scheduler</h3>
  <div style={{ padding: '5px' }}>
    <p>
      Before we mostly relied on CloudWatch Event rules to schedule tasks it
      worked, but it was limited no one-time schedules, no payload flexibility,
      and not great for managing thousands of tasks.
    </p>
    <p>EventBridge Scheduler changes the game:</p>
    <ol>
      <li>It can handle millions of schedules</li>
      <li>Supports both one-time and recurring invocations</li>
      <li>Invokes over 270 AWS services or even external APIs</li>
      <li>Has fine-grained retry, encryption, and delivery window controls</li>
      <li>And fully serverless Zero cron servers, Zero EC2 instances</li>
    </ol>
    
  <img src="/images/posts/event-bridge-scheduler/1.png" />
  <p>Amazon EventBridge Scheduler offers key features that make scheduling <strong>simple</strong> and <strong>reliable</strong>. We can use <strong>templated targets</strong> for quick setup of common operations, or <strong>universal targets</strong> for custom triggers across AWS services. <strong>Flexible time windows</strong> let us spread out schedules when exact timing isn't critical, while built-in <strong>retries</strong> ensure our tasks execute successfully even if there are temporary failures.</p>

  </div>
</div>

<div style={{ marginTop: '20px' }}>
  <h3>Schedule Types</h3>
  <div style={{ padding: '5px' }}>
    <p>
      When we're setting up schedules in EventBridge Scheduler, we have the
      flexibility to define them using three different expression types as
      showing in the table bellow
    </p>
    <table>
      <tr>
        <th>Type</th>
        <th>Example</th>
        <th>Description</th>
      </tr>
      <tr>
        <td>at(2025-12-01T10:00:00Z)</td>
        <td>one-time</td>
        <td>Run exactly once</td>
      </tr>
      <tr>
        <td>cron(0 12 * * ? *)</td>
        <td>every day at 12:00 UTC</td>
        <td>Full cron power</td>
      </tr>
      <tr>
        <td>rate(5 minutes)</td>
        <td>every 5 minutes</td>
        <td>Simple repetition</td>
      </tr>
    </table>
    <p>
      This gives us the control to choose the scheduling method that best fits
      our specific use case, throughout the rest of this post, we'll explore
      each of these expression types in detail.
    </p>
  </div>
</div>

<div style={{ marginTop: '20px' }}>
  <h3>Create EventBridge Schedule</h3>
  <div style={{ padding: '5px' }}>
    <p>
       When creating a schedule, vai the console after entering the name and description, we need to choose the
      schedule pattern as shown in the image below
    </p>
    <img src="/images/posts/event-bridge-scheduler/2.png" />
    <p>
      As we can see, we have 2 patterns: one-time schedule and recurring
      schedule (cron + rate), for the one-time option, it's simple we just need
      to specify the date, time, time zone, and the flexible time window
    </p>
    
    <aside
      style={{
        borderLeft: '4px solid #2b6cb0',
        padding: '12px',
        background: '#f1f5f9',
        width: '100%',
      }}
    >
      <strong>Note:</strong>
      <p style={{ margin: '8px 0 0' }}>
        The flexible time window allows the schedule to execute within a range
        of time rather than at an exact moment. This helps improve reliability
        and distribute load across your system
      </p>
    </aside>
<p>
  For the recurring option, we need to choose between a cron-based schedule or a
  rate-based schedule, then enter the corresponding cron expression or rate
  expression.
</p>
  </div>

</div>

<img
  src="/images/posts/event-bridge-scheduler/3.png"
  alt="cron-based schedule"
/>

<img
  src="/images/posts/event-bridge-scheduler/4.png"
  alt="rate-based schedule"
/>

<div style={{ padding: '5px' }}>
  <p>
    The next step is selecting a target. In this case, I've chosen an AWS Lambda
    function to demonstrate the process
  </p>
</div>

<img src="/images/posts/event-bridge-scheduler/5.png" alt="Schedule Target" />

<div style={{ padding: '5px' }}>
  <p>
    Then we select the Lambda function we want to trigger. We can also specify a
    JSON payload, which can be helpful in certain situations where we need to
    pass specific data to our function
  </p>
  <img
    src="/images/posts/event-bridge-scheduler/6.png"
    alt="Function and Payload"
  />

  <p>
    With that, our configuration is complete and the schedule is ready to execute. 
    keep in mind that the scheduler needs permission to invoke our function, which is handled 
    automatically by AWS when we create the schedule through the console.
  </p>
</div>

<div style={{ marginTop: '20px' }}>
  <h3>Using the Node.js SDK</h3>
  <div style={{ padding: '5px' }}>
    In this section, we'll walk through creating and configuring an EventBridge
    schedule using the Node.js SDK, first, we need to install the AWS SDK
    package also we need to define a global client that will allow us to
    interact with the EventBridge API
  </div>
</div>

```javascript
  // install the sdk
  npm install @aws-sdk/client-scheduler

  // define the client
  const client = new SchedulerClient({ region: "us-east-1" });
```

<div style={{ marginTop: '20px' }}>
  <h4>One-Time Schedule</h4>
  <div style={{ padding: '5px' }}>
    If we need to trigger a function just once at a specific date and time we
    can use the <strong>at()</strong> syntax.
  </div>
</div>

```javascript
async function scheduleOrderProcessing(orderId, runAt) {
  const command = new CreateScheduleCommand({
    Name: `process-order-${orderId}`,
    ScheduleExpression: `at(${runAt})`,
    FlexibleTimeWindow: { Mode: 'OFF' },
    Target: {
      Arn: 'arn:aws:lambda:us-east-1:123456789012:function:processOrder',
      RoleArn: 'arn:aws:iam::123456789012:role/EventBridgeSchedulerRole',
      Input: JSON.stringify({ orderId }),
    },
  });
  const response = await client.send(command);
  console.log(`📦 Schedule created for order ${orderId} at ${runAt}`);
}
```

<div style={{ padding: '5px' }}>
  <p>
    This function creates a one-time EventBridge schedule to trigger a Lambda
    function at a specific time. It takes an <strong>orderId</strong> and{' '}
    <strong>runAt</strong> timestamp as parameters, then uses the{' '}
    <strong>CreateScheduleCommand</strong> to set up the schedule with a unique
    name based on the order ID. The schedule uses the <strong>at()</strong>{' '}
    expression for one-time execution, targets a specific Lambda function with
    the necessary IAM role permissions, and passes the <strong>orderId</strong>{' '}
    as JSON input through the <strong>Input</strong> field. This payload will be
    available to our Lambda function when it executes, allowing it to know which
    order to process
  </p>
</div>

<div style={{ marginTop: '20px' }}>
  <h4>Cron-based Schedule</h4>
</div>

```javascript
async function createDailyReportSchedule() {
  const command = new CreateScheduleCommand({
    Name: 'daily-report',
    Description: 'Generate a daily report at 6 AM Tunis time',
    ScheduleExpression: 'cron(0 6 * * ? *)',
    ScheduleExpressionTimezone: 'Europe/Tunis',
    FlexibleTimeWindow: { Mode: 'OFF' },
    Target: {
      Arn: 'arn:aws:lambda:us-east-1:123456789012:function:generateDailyReport',
      RoleArn: 'arn:aws:iam::123456789012:role/EventBridgeSchedulerRole',
      Input: JSON.stringify({ type: 'daily' }),
    },
  });

  const response = await client.send(command);
  console.log('✅ Schedule created:', response);
}
```

<div style={{ padding: '5px' }}>
  <p>
    This function creates a recurring schedule that runs every day at 6 AM using
    a cron expression. It triggers a Lambda function to generate reports and
    passes type: "daily" as input to specify the report type.
  </p>
</div>

<div style={{ marginTop: '20px' }}>
  <h4>Rate-based Schedule</h4>
  <p>
    For the rate-based scheduler, it's the same process. We just need to change
    the ScheduleExpression from a cron expression to a rate expression
  </p>
</div>

```javascript
async function createHourlyReportSchedule() {
  const command = new CreateScheduleCommand({
    Name: 'hourly-report',
    Description: 'Generate a report every hour',
    ScheduleExpression: 'rate(1 hour)',
    FlexibleTimeWindow: { Mode: 'OFF' },
    Target: {
      Arn: 'arn:aws:lambda:us-east-1:123456789012:function:generateDailyReport',
      RoleArn: 'arn:aws:iam::123456789012:role/EventBridgeSchedulerRole',
      Input: JSON.stringify({ type: 'hourly' }),
    },
  });
  const response = await client.send(command);
  console.log('✅ Schedule created:', response);
}
```

<div style={{ marginTop: '20px' }}>
  <h3>Retry policy, DLQ and Reliability</h3>
  <p>
    EventBridge Scheduler allows us to configure retry behavior for failed
    invocations and we can customize:
  </p>
  <ol>
    <li>Max attempts</li>
    <li>Backoff strategy</li>
    <li>Dead Letter Queue (DLQ) for debugging</li>
  </ol>
  <img
    src="/images/posts/event-bridge-scheduler/7.png"
    alt="Retry policy and dead-letter queue (DLQ)"
  />
  <p>
    This ensures our tasks are idempotent, meaning even if the same schedule
    runs twice due to retries, our logic should handle it safely.
  </p>
  <p>
    <strong>Best practice:</strong> Use a lightweight Lambda as a trigger and
    enqueue jobs into SQS or Step Functions for the real work. That way, the
    Scheduler never overloads our main services.
  </p>
</div>

<div style={{ marginTop: '20px' }}>
  <h3>Common Patterns</h3>
  <p>Here are a few ways teams use it in production:</p>
  <ol>
    <li>
      <strong>Daily report generation:</strong> Scheduler → Lambda → S3 → SNS
      notification
    </li>
    <li>
      <strong>Time-based workflow orchestration:</strong> Scheduler → Step
      Functions → Multi-step business flow
    </li>
    <li>
      <strong>Webhook / API calling:</strong> Scheduler → API Destination →
      External system
    </li>
    <li>
      <strong>Deferred retries:</strong> Scheduler → one-time "retry later"
      Lambda trigger
    </li>
  </ol>
</div>

<div style={{ marginTop: '20px' }}>
  <h3>Important Considerations</h3>
  <ol>
    <li>
      If we use very frequent schedules (like every minute for thousands of
      items), costs can pile up fast consider batching.
    </li>
    <li>
      Each schedule name must be unique. Use IDs or timestamps in names for
      one-time jobs.
    </li>
    <li>
      Cron timezones can be tricky always test with UTC first, then apply{' '}
      <strong>ScheduleExpressionTimezone</strong>.
    </li>
    <li>Delete schedules we no longer need they persist until removed.</li>
  </ol>
</div>

<div style={{ marginTop: '20px' }}>
  <h3>Conclusion</h3>
   <p>
   EventBridge Scheduler is one of those AWS services that feels like a small feature but solves a huge pain. 
   No more EC2 cron daemons, no more Lambda self-invocation hacks, no more <b>sleep queue</b> patterns
</p>

  <p>
  It's clean, scalable, and beautifully simple the kind of AWS service we can just <b>set and forget</b>
  </p>
</div>


EventBridge Scheduler


<div style={{ marginTop: '20px' }}>
  <h3>Introduction</h3>
  <div style={{ padding: '5px' }}>
    <p>
      For those coming from a relational database background, the idea of
      placing multiple entity types in a single DynamoDB table can initially
      seem counterintuitive. Keep in mind that DynamoDB is a table service, not
      a traditional database service. In practice, however, the{' '}
      <strong>one-table design pattern</strong> is not just a best practice it
      is recommended by AWS itself and often serves as the foundation for
      building scalable, cost-efficient, and high-performance applications on
      DynamoDB.
    </p>
  </div>
</div>

<div style={{ marginTop: '20px' }}>
  <h3>One-Table Pattern Explained</h3>
  <div style={{ padding: '5px' }}>
  <p>
  The <strong>one-table pattern</strong> is a data modeling technique that stores all of an application's entities such as users, orders, products, and comments in a single DynamoDB table. 
  Unlike relational databases, which create separate tables for each type of entity, this method relies on a flexible schema using partition keys (PK), sort keys (SK), and smart use of Global Secondary Indexes (GSIs) to make everything work efficiently.
</p>

<p style={{ marginTop: '5px' }}>
  The <strong>one-table design</strong> offers significant advantages for
  building efficient and manageable applications. Storing multiple entity types
  in a single DynamoDB table improves performance, lowers costs, and simplifies
  operations, creating a system that is both scalable and resilient.
</p>

<h6 style={{ marginTop: '10px' }}>Performance Benefits:</h6>
<ul>
  <li>Reduced latency by fetching related data in single queries</li>
  <li>Ability to retrieve multiple entity types in one request</li>
  <li>Efficient use of DynamoDB's querying capabilities</li>
</ul>

<h6>Cost Efficiency:</h6>
<ul>
  <li>Fewer tables mean lower baseline costs</li>
  <li>Better utilization of provisioned capacity</li>
  <li>Reduced overhead for on-demand billing</li>
</ul>

<h6>Operational Simplicity:</h6>
<ul>
  <li>Fewer tables to manage and monitor</li>
  <li>Simplified backup and restore procedures</li>
  <li>Easier capacity planning</li>
</ul>
  </div>
</div>

<div style={{ marginTop: '20px' }}>
  <h3>Core Concepts</h3>
  <div style={{ padding: '5px' }}>
  <p>
  Before exploring the implementation, it's important to understand a few key concepts that will provide the foundation for the next sections.
</p>
  <h4 style={{marginTop: "10px"}}>Primary Key Design</h4>
<p>
  In DynamoDB, every table requires a primary key that uniquely identifies each item. 
  When using a <strong>composite primary key</strong>, it consists of two parts:
</p>

<ul>
  <li>
    <strong>Partition Key (PK):</strong> Determines how data is distributed
    across storage partitions. Items with the same partition key are stored
    together, which makes related data easier to query efficiently.
  </li>
  <li>
    <strong>Sort Key (SK):</strong> Defines the order of items within a
    partition and allows for sorting and range-based queries. This makes it
    possible to fetch related records quickly without scanning the entire table.
  </li>
</ul>
<h4 style={{ marginTop: '10px' }}>Generic Attribute Names</h4>
<p>
  When storing multiple types of items in a single table (such as users, orders,
  and products), it is better to avoid specific key names like{' '}
  <strong>userId</strong> or <strong>orderId</strong>.
</p>

<p>
  Generic names such as <strong>PK</strong> (Partition Key) and{' '}
  <strong>SK</strong> (Sort Key) should be used instead. For secondary indexes,
  common names like <strong>GSI1PK</strong> and <strong>GSI1SK</strong> are
  typically used.
</p>

<p>
  This naming approach allows various entity types to coexist within the same
  table, as the key values themselves define what each item represents.
</p>

<p>
  <strong>Example:</strong>
</p>
<ul>
  <li>User item → PK = USER#123, SK = METADATA</li>
  <li>Order item → PK = USER#123, SK = ORDER#456</li>
</ul>

<aside
  style={{
    borderLeft: '4px solid #16a34a',
    padding: '12px',
    background: '#f0fdf4',
    width: '100%',
    borderRadius: '6px',
  }}
>
  <strong>In short</strong>
  <p style={{ margin: '8px 0 0' }}>
    Using generic keys keeps the table design flexible and adaptable for
    different entities
  </p>
</aside>

<h4 style={{ marginTop: '10px' }}>Hierarchical Data</h4>
<p>
  Related entities can be grouped under the same{' '}
  <strong>Partition Key (PK)</strong>. This makes it possible to fetch an entire
  relationship like a parent and all its children in a single query.
</p>

<p>
  For example, consider a system that stores users and their orders. All items
  related to the same user share the same <strong>PK</strong>, but have
  different <strong>Sort Keys (SK)</strong>.
</p>

<ul>
  <li>User profile → PK = USER#123, SK = METADATA</li>
  <li>Order 1 → PK = USER#123, SK = ORDER#1</li>
  <li>Order 2 → PK = USER#123, SK = ORDER#2</li>
  <li>Order 3 → PK = USER#123, SK = ORDER#3</li>
</ul>

<p>
  With this structure, a single query on <strong>PK = USER#123</strong>
  retrieves the user profile and all orders at once.
</p>

<aside
  style={{
    borderLeft: '4px solid #16a34a',
    padding: '12px',
    background: '#f0fdf4',
    width: '100%',
    borderRadius: '6px',
  }}
>
  <strong>In short</strong>
  <p style={{ margin: '8px 0 0' }}>
    Storing related entities under the same partition key groups them naturally.
    This structure allows retrieving all connected items ike a user and their
    orders with a single efficient query
  </p>
</aside>

  </div>
</div>

<div style={{ marginTop: '20px' }}>
  <h3>Implementation</h3>
  <div style={{ padding: '5px' }}>
 <p>
  In this section, we will implement the one-table pattern through the example of Users, Products and Orders
   using Nodejs with the <strong>AWS SDK</strong> to interact with <strong>DynamoDB</strong> but the same logic can be applied using any other supported programming language.
</p>

<h4 style={{ marginTop: '10px' }}>Setting Up the Project</h4>

<p>
  First, let's start by defining two global variables that we will use later
  <strong>client</strong> and <strong>docClient</strong> and the table name in our
  case AppTable as it is a generic name
</p>

<aside
  style={{
    borderLeft: '4px solid #2b6cb0',
    padding: '12px',
    background: '#f1f5f9',
    width: '100%',
  }}
>
  <strong>Note:</strong>
  <p style={{ margin: '8px 0 0' }}>
    I have already created a DynamoDB table called AppTable via the console, as shown in the image below.

  </p>
</aside>

<img src="/images/posts/dynamo-db-one-table/1.png" alt="AppTable" />

```javascript
import { DynamoDBClient } from '@aws-sdk/client-dynamodb';
import {
  DynamoDBDocumentClient,
  PutCommand,
  QueryCommand,
  GetCommand,
} from '@aws-sdk/lib-dynamodb';

const client = new DynamoDBClient({ region: 'eu-central-1' });
const docClient = DynamoDBDocumentClient.from(client);

const TABLE_NAME = 'AppTable';
```

<p>
  The <strong>client</strong> is the low-level DynamoDB client that communicates
  directly with DynamoDB using raw data formats.
</p>
<p>
  The <strong>docClient</strong> is a higher-level wrapper around{' '}
  <strong>client</strong> that allows working with plain JavaScript objects for
  easier DynamoDB operations.
</p>

<h4 style={{ marginTop: '10px' }}>Designing the Key Structure</h4>
<p>Here's how we'll structure our keys:</p>

```javascript
## Users
PK: USER#<userId>
SK: METADATA

## Orders
PK: USER#<userId>
SK: ORDER#<orderId>

## Products
PK: PRODUCT#<productId>
SK: METADATA

## Order Items
PK: ORDER#<orderId>
SK: PRODUCT#<productId>

```

<h4 style={{ marginTop: '10px' }}>Creating Entities</h4>
<p>
  Now we are ready to start defining functions that will be used to create
  entities in the table, beginning with the main entity <strong>user</strong>.
</p>

```javascript
async function createUser(userId, userData) {
  const params = {
    TableName: TABLE_NAME,
    Item: {
      PK: `USER#${userId}`,
      SK: 'METADATA',
      EntityType: 'User',
      userId,
      email: userData.email,
      name: userData.name,
      createdAt: new Date().toISOString(),
      // GSI for email lookup
      GSI1PK: `EMAIL#${userData.email}`,
      GSI1SK: `USER#${userId}`,
    },
  };

  await docClient.send(new PutCommand(params));
  return params.Item;
}
```

<p>
  The <strong>createUser</strong> function creates a new user item in the
  DynamoDB table. It takes a <code>userId</code> and <code>userData</code> (like{' '}
  <code>email</code> and <code>name</code>) and constructs an item with a{' '}
  <strong>partition key (PK)</strong>, <strong>sort key (SK)</strong>, and other
  attributes. It also sets up a <strong>global secondary index (GSI)</strong>{' '}
  for looking up users by email. Finally, it uses{' '}
  <code>docClient.send(new PutCommand(params))</code> to insert the item into
  the table and returns the created item. We could add more fields like{' '}
  <code>password</code> or <code>role</code>, but for the sake of this post, we
  are using only simple attributes.
</p>

<aside
  style={{
    borderLeft: '4px solid #2b6cb0',
    padding: '12px',
    background: '#f1f5f9',
    width: '100%',
  }}
>
  <strong>Note:</strong>
  <p style={{ margin: '8px 0 0' }}>
    We used GSI1PK and GSI1SK so we can look up users by email, but just adding
    these fields in the item is not enough. We also need to set up the GSI in
    the table settings so DynamoDB knows to index and use them for fast queries.
  </p>
</aside>

```javascript
// Product Management
async function createProduct(productId, productData) {
  const params = {
    TableName: TABLE_NAME,
    Item: {
      PK: `PRODUCT#${productId}`,
      SK: 'METADATA',
      EntityType: 'Product',
      productId,
      name: productData.name,
      price: productData.price,
      category: productData.category,
      // GSI for category queries
      GSI1PK: `CATEGORY#${productData.category}`,
      GSI1SK: `PRODUCT#${productId}`,
    },
  };

  await docClient.send(new PutCommand(params));
  return params.Item;
}
```

<p>
  The <strong>createProduct</strong> function adds a new product item to the
  DynamoDB table. It uses <code>productId</code> as the primary key and stores
  details like <code>name</code>, <code>price</code>, and <code>category</code>.
  A GSI is also defined with <code>GSI1PK</code> and <code>GSI1SK</code> to
  allow efficient querying of products by category. The function inserts the
  item into the table and returns the created product.
</p>

```javascript
async function createOrder(userId, orderId, orderData) {
  const params = {
    TableName: TABLE_NAME,
    Item: {
      PK: `USER#${userId}`,
      SK: `ORDER#${orderId}`,
      EntityType: 'Order',
      orderId,
      userId,
      status: orderData.status,
      total: orderData.total,
      createdAt: new Date().toISOString(),
      // GSI for querying orders by status
      GSI1PK: `ORDERSTATUS#${orderData.status}`,
      GSI1SK: orderId,
    },
  };

  await docClient.send(new PutCommand(params));
  return params.Item;
}
```

<p>
  The <strong>createOrder</strong> function creates a new order item in the
  DynamoDB table. It uses <code>userId</code> and <code>orderId</code> to build
  the primary key and stores details like <code>status</code>,{' '}
  <code>total</code>, and creation date. It also sets up a GSI with{' '}
  <code>GSI1PK</code> and <code>GSI1SK</code> to allow efficient querying of
  orders by their status. The function inserts the item into the table and
  returns the created order.
</p>

<p>
  To see the beauty of this pattern, let's orchestrate all these functions in
  one place and run them.
</p>

```javascript
async function main() {
  await createUser(1, { email: 'test@gmail.com', name: 'test' });

  await createProduct(101, {
    name: 'Laptop',
    price: 1200,
    category: 'Electronics',
  });

  await createOrder(1, 101, { status: 'PENDING', total: 250 });
}

main();
```

<img
  src="/images/posts/dynamo-db-one-table/3.png"
  alt="Result from the UI console"
/>

<h4 style={{ marginTop: '10px' }}>Querying Patterns</h4>

<p>
  The real advantage of this pattern is the power and flexibility it provides
  for querying. The code snippets below illustrate several use cases based on
  our example.
</p>

```javascript
// Get user and all their orders in ONE query
async function getUserWithOrders(userId) {
  const params = {
    TableName: TABLE_NAME,
    KeyConditionExpression: 'PK = :pk',
    ExpressionAttributeValues: {
      ':pk': `USER#${userId}`,
    },
  };

  const result = await docClient.send(new QueryCommand(params));

  const user = result.Items.find((item) => item.SK === 'METADATA');
  const orders = result.Items.filter((item) => item.SK.startsWith('ORDER#'));

  return { user, orders };
}
```

<img
  src="/images/posts/dynamo-db-one-table/4.png"
  alt="Response of the getUserWithOrders"
/>

```javascript
// Find user by email using GSI
async function getUserByEmail(email) {
  const params = {
    TableName: TABLE_NAME,
    IndexName: 'GSI1',
    KeyConditionExpression: 'GSI1PK = :gsi1pk',
    ExpressionAttributeValues: {
      ':gsi1pk': `EMAIL#${email}`,
    },
  };

  const result = await docClient.send(new QueryCommand(params));
  return result.Items[0];
}

// Get all orders with a specific status
async function getOrdersByStatus(status) {
  const params = {
    TableName: TABLE_NAME,
    IndexName: 'GSI1',
    KeyConditionExpression: 'GSI1PK = :gsi1pk',
    ExpressionAttributeValues: {
      ':gsi1pk': `ORDERSTATUS#${status}`,
    },
  };

  const result = await docClient.send(new QueryCommand(params));
  return result.Items;
}
```

<p>
  These functions show how to <strong>query items using GSIs</strong> in the
  DynamoDB table.
</p>
<p>
  <strong>getUserByEmail(email)</strong>: Queries the <code>GSI1</code> index to
  find a user by their email and returns the first matching item.
</p>
<p>
  <strong>getOrdersByStatus(status)</strong>: Queries the same index to retrieve
  all orders with a specific status and returns the matching items.
</p>
<p>
  Both functions use the GSI partition key to efficiently fetch items without
  scanning the whole table.
</p>

```javascript
// Get products by category
async function getProductsByCategory(category) {
  const params = {
    TableName: TABLE_NAME,
    IndexName: 'GSI1',
    KeyConditionExpression: 'GSI1PK = :gsi1pk',
    ExpressionAttributeValues: {
      ':gsi1pk': `CATEGORY#${category}`,
    },
  };

  const result = await docClient.send(new QueryCommand(params));
  return result.Items;
}
```

<p>
  The <strong>getProductsByCategory(category)</strong> function queries the{' '}
  <code>GSI1</code> index to retrieve all products within a specific category.
  It uses the GSI partition key to efficiently fetch items without scanning the
  entire table and returns the matching products.
</p>

</div>
</div>

<div style={{ marginTop: '20px' }}>
  <h3>Performance Optimization</h3>
  <div style={{ padding: '5px' }}>
  <p>
This section will cover some performance optimizations that will help improve DynamoDB queries in terms of speed and cost.
</p>

   <h4>Batch Operations</h4>
   Use batch operations for efficiency:

```javascript
import { BatchWriteCommand } from '@aws-sdk/lib-dynamodb';

async function batchCreateProducts(products) {
  const chunks = chunkArray(products, 25); // DynamoDB batch limit

  for (const chunk of chunks) {
    const params = {
      RequestItems: {
        [TABLE_NAME]: chunk.map((product) => ({
          PutRequest: {
            Item: {
              PK: `PRODUCT#${product.id}`,
              SK: 'METADATA',
              ...product,
            },
          },
        })),
      },
    };

    await docClient.send(new BatchWriteCommand(params));
  }
}
```

<h4>Projection Expressions</h4>
Only retrieve the needed attributes:

```javascript
async function getUserEmail(userId) {
  const params = {
    TableName: TABLE_NAME,
    Key: { PK: `USER#${userId}`, SK: 'METADATA' },
    ProjectionExpression: 'email, #n',
    ExpressionAttributeNames: { '#n': 'name' },
  };

  const result = await docClient.send(new GetCommand(params));
  return result.Item;
}
```

<h4>Consistent Reads</h4>
Use consistent reads only when necessary, as they cost twice as much:

```javascript
const params = {
  TableName: TABLE_NAME,
  Key: { PK: `USER#${userId}`, SK: 'METADATA' },
  ConsistentRead: true, // Be careful with this
};
```

</div>
</div>

<div style={{ marginTop: '20px' }}>
  <h3>Conclusion</h3>
  <div style={{ padding: '5px' }}>
    <p>
      The DynamoDB one-table pattern is a paradigm shift from traditional
      database design, but it unlocks the full power of DynamoDB's performance
      and scalability. By carefully planning your access patterns, using generic
      key names, and leveraging GSIs strategically, we can build highly
      efficient applications that scale seamlessly.
    </p>
  </div>
</div>


DynamoDB One Table Pattern


<div style={{ marginTop: '20px' }}>
  <h3>Introduction</h3>
  <div style={{ padding: '5px' }}>
    <p>
      In modern serverless or event-driven architectures, data changes should
      not exist in isolation. Typically, these changes need to trigger reactions
      such as updates, data synchronization or notifications.
    </p>
    <p style={{ marginTop: '20px' }}>
      DynamoDB Streams captures every <strong>change</strong> that happens to
      items in any DynamoDB table and makes that data available{' '}
      <strong>almost instantly</strong>. We can process these changes using{' '}
      <strong>Lambda functions</strong>, <strong>Kinesis</strong>, or even build
      our own custom consumers with <strong>Node.js</strong>.
    </p>
  </div>
</div>

<div style={{ marginTop: '20px' }}>
  <h3>DynamoDB Streams Explained</h3>
  <div style={{ padding: '5px' }}>
    <p>
      DynamoDB Streams is a time-ordered sequence of item-level changes (insert,
      update, remove) in a table.
    </p>
    <img src="/images/posts/lambda-invocations/1.png" alt="DynamoDB Streams" />
    <p>
      Each stream record represents a <strong>modification event</strong> and
      contains:
    </p>
    <ul>
      <li>
        The type of operation (<strong>INSERT</strong>, <strong>MODIFY</strong>,{' '}
        <strong>REMOVE</strong>)
      </li>
      <li>
        The <strong>keys</strong> of the modified item
      </li>
      <li>
        The <strong>old</strong> and/or <strong>new</strong> image of the item
        (depending on stream view type)
      </li>
      <li>
        Metadata like <strong>sequence number</strong> and{' '}
        <strong>event time</strong>
      </li>
    </ul>

<p>We can configure a stream to capture:</p>

<table>
  <thead>
    <tr>
      <th>Stream View Type</th>
      <th>Description</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><strong>KEYS_ONLY</strong></td>
      <td>Only primary key attributes</td>
    </tr>
    <tr>
      <td><strong>NEW_IMAGE</strong></td>
      <td>Entire item after modification</td>
    </tr>
    <tr>
      <td><strong>OLD_IMAGE</strong></td>
      <td>Entire item before modification</td>
    </tr>
    <tr>
      <td><strong>NEW_AND_OLD_IMAGES</strong></td>
      <td>Both old and new versions</td>
    </tr>
  </tbody>
</table>
  </div>
</div>

<div style={{ marginTop: '20px' }}>
  <h3>Configure and enable streams</h3>
  <div style={{ padding: '5px' }}>
    <p>
      To use DynamoDB Streams, we first need to <strong>enable streams</strong>{' '}
      on our table.
    </p>
    <img
      src="/images/posts/dynamodb-streams/1.png"
      alt="Enable streams from the console"
    />
    <p>
      To enable streams, go to the <strong>DynamoDB console</strong> and select
      your table. Navigate to the <strong>Exports and streams</strong> tab, then
      click <strong>Enable</strong> under DynamoDB stream details. Choose your
      preferred <strong>stream view type</strong> (KEYS_ONLY, NEW_IMAGE,
      OLD_IMAGE, or NEW_AND_OLD_IMAGES) and click <strong>Enable stream</strong>{' '}
      to confirm. Alternatively, you can use the <strong>AWS CLI</strong> with
      the <strong>update-table</strong> command and the{' '}
      <strong>--stream-specification</strong> parameter to enable streams
      programmatically.
    </p>
    <img
      src="/images/posts/dynamodb-streams/2.png"
      alt="Stream enabled for doodoot-table"
    />
  
<p>
  To actually use DynamoDB Streams with Lambda, Kinesis or manual NodeJS Consumer, we need to set up
  some configuration. Here's what each integration requires:
</p>

<h4 style={{ marginTop: '5px' }}>Connecting to Lambda</h4>
<p>
  We need to create an <strong>event source mapping</strong> that links our
  DynamoDB Stream to our Lambda function. Our Lambda function also needs{' '}
  <strong>IAM permissions</strong> to read from the stream—things like{' '}
  <strong>dynamodb:GetRecords</strong>,{' '}
  <strong>dynamodb:GetShardIterator</strong>, and{' '}
  <strong>dynamodb:DescribeStream</strong>. We can also configure settings like{' '}
  <strong>batch size</strong>, <strong>starting position</strong>, and how to
  handle errors.
</p>

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:GetRecords",
        "dynamodb:GetShardIterator",
        "dynamodb:DescribeStream",
        "dynamodb:ListStreams"
      ],
      "Resource": "arn:aws:dynamodb:REGION:ACCOUNT_ID:table/TABLE_NAME/stream/*"
    }
  ]
}
```

<h4 style={{ marginTop: '5px' }}>Connecting to Kinesis Data Streams</h4>
<p>
  For Kinesis, we'll enable <strong>Kinesis Data Streams for DynamoDB</strong>,
  which is actually a separate feature from DynamoDB Streams. We configure the
  Kinesis stream as a destination, and we'll need{' '}
  <strong>IAM permissions</strong> that allow DynamoDB to write to our Kinesis
  stream.
</p>

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:PutRecord",
        "kinesis:PutRecords",
        "kinesis:DescribeStream"
      ],
      "Resource": "arn:aws:kinesis:REGION:ACCOUNT_ID:stream/STREAM_NAME"
    }
  ]
}
```

<h4 style={{ marginTop: '5px' }}>Building a Custom Consumer</h4>
<p>
  If we're building our own consumer with Node.js, we'll need the{' '}
  <strong>AWS SDK</strong> installed. Our application needs{' '}
  <strong>IAM credentials</strong> with the right permissions to read from
  DynamoDB Streams, and we'll manually poll the stream using SDK methods.
</p>

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:GetRecords",
        "dynamodb:GetShardIterator",
        "dynamodb:DescribeStream",
        "dynamodb:ListStreams"
      ],
      "Resource": "arn:aws:dynamodb:REGION:ACCOUNT_ID:table/TABLE_NAME/stream/*"
    }
  ]
}
```

<p>All these integrations require some setup mainly around <strong>permissions</strong> and <strong>connecting the services together</strong>.</p>
  </div>
</div>

<div style={{ marginTop: '20px' }}>
  <h3>Architectural Patterns</h3>
  <div style={{ padding: '5px' }}>
  <p>DynamoDB Streams enables several powerful architectural patterns that help us build scalable and event-driven applications. Here are some common patterns we can implement:</p>

<h4 style={{ marginTop: '5px' }}>Event-Driven Architecture</h4>
<p>
  We can use DynamoDB Streams to trigger <strong>Lambda functions</strong>{' '}
  automatically whenever data changes in our table. This allows us to build
  reactive systems where different parts of our application respond to data
  changes without tight coupling.
</p>

<h4 style={{ marginTop: '5px' }}>Data Replication</h4>
<p>
  Streams make it easy to <strong>replicate data</strong> across different
  tables, regions, or even different databases. We can capture changes in one
  table and automatically propagate them to another location, ensuring data
  consistency across our infrastructure.
</p>

<h4 style={{ marginTop: '5px' }}>Real-Time Analytics</h4>
<p>
  By connecting DynamoDB Streams to <strong>Kinesis Data Streams</strong>, we
  can send change data to analytics pipelines for real-time processing. This is
  useful for generating dashboards, tracking metrics, or detecting patterns as
  data changes.
</p>

<h4 style={{ marginTop: '5px' }}>Audit Logging</h4>
<p>
  We can capture every change made to our DynamoDB table and write it to a{' '}
  <strong>logging system</strong> or an <strong>S3 bucket</strong>. This creates
  a complete audit trail of all modifications, which is essential for compliance
  and debugging.
</p>

<h4 style={{ marginTop: '5px' }}>Cache Invalidation</h4>
<p>
  When data in DynamoDB changes, we can use Streams to automatically{' '}
  <strong>invalidate or update caches</strong> (like ElastiCache or CloudFront).
  This ensures our cached data stays in sync with the source of truth.
</p>

<h4 style={{ marginTop: '5px' }}>Aggregation and Materialized Views</h4>
<p>
  We can use Streams to maintain <strong>aggregated data</strong> or{' '}
  <strong>materialized views</strong> in separate tables. For example, when
  individual sales records are added, we can automatically update summary tables
  with totals and statistics.
</p>

<h4 style={{marginTop: "5px"}}>Cross-Service Communication</h4>
<p>Streams act as a bridge between DynamoDB and other AWS services. We can trigger <strong>Step Functions workflows</strong>, send notifications via <strong>SNS</strong>, or invoke other microservices based on database changes.</p>
  </div>
 <img
      src="/images/posts/dynamodb-streams/3.png"
      alt="Architecture Overview"
    />
</div>

<div style={{ marginTop: '20px' }}>
  <h3> Connecting to Lambda function</h3>
  <div style={{ padding: '5px' }}>

<p>
  To connect DynamoDB Streams with Lambda, we first need to make sure{' '}
  <strong>streams are enabled</strong> on our DynamoDB table with the
  appropriate <strong>stream view type</strong>. Once that's configured, we can
  set up the Lambda trigger:
</p>

<ul>
  <li>
    Go to the <strong>AWS Lambda console</strong> and either create a new
    function or select an existing one.
  </li>
  <li>
    Navigate to the <strong>Configuration</strong> tab and click on{' '}
    <strong>Triggers</strong>.
  </li>
  <li>
    Select <strong>Add trigger</strong> and choose <strong>DynamoDB</strong> as
    the trigger source.
  </li>
  <li>Select your table from the dropdown menu.</li>
  <li>
    Configure the <strong>batch size</strong> (how many records Lambda processes
    at once) and <strong>starting position</strong> (LATEST for new changes or
    TRIM_HORIZON for existing records).
  </li>
  <li>
    Optionally, set up error handling options like{' '}
    <strong>retry attempts</strong> and <strong>failure destinations</strong>.
  </li>
  <li>
    Save the trigger, and Lambda will automatically start polling your DynamoDB
    Stream and invoking your function whenever new records are available.
  </li>
</ul>

<img
  src="/images/posts/dynamodb-streams/6.png"
  alt="Added Lambda as a trigger"
/>

<aside
  style={{
    borderLeft: '4px solid #2b6cb0',
    padding: '12px',
    background: '#f1f5f9',
    width: '100%',
  }}
>
  <strong>Note:</strong>
  <p style={{ margin: '8px 0 0' }}>
    When adding Lambda as a trigger, we can specify the{' '}
    <strong>batch size</strong>, which defines how many stream records the
    function will process in a single invocation. This helps optimize
    performance and control costs, as shown in the image below.
  </p>
</aside>

<img src="/images/posts/dynamodb-streams/5.png" alt="Creating the trigger" />

<p>
  To handle the records coming from the stream, we can use the following code to
  iterate through each record and perform actions like forwarding them to
  downstream services such as <strong>S3</strong>, <strong>SNS</strong>, or{' '}
  <strong>SQS</strong>.
</p>

```javascript
export const handler = async (event) => {
  for (const record of event.Records) {
    console.log('Event:', record.eventName);
    console.log('Keys:', record.dynamodb.Keys);
  }
};
```

<p>
  The stream can also be used to automatically replicate data to another
  DynamoDB table in a different region. Whenever an item is inserted, updated,
  or deleted in the source table, our consumer captures that change and writes
  it to the destination table, creating a <strong>cross-region backup</strong>{' '}
  or <strong>multi-region setup</strong>.
</p>

```javascript
if (record.eventName === 'MODIFY') {
  await replicaClient.putItem({
    TableName: 'doodootiReplica',
    Item: record.dynamodb.NewImage,
  });
}
```

  </div>
</div>

<div style={{ marginTop: '20px' }}>
  <h3>Connecting to Kinesis Data Streams</h3>
  <div style={{ padding: '5px' }}>
<p>DynamoDB can directly replicate stream data to <strong>Kinesis Data Streams</strong> using the Kinesis Data Streams for DynamoDB feature. This integration allows us to fan out stream records to multiple consumers and leverage Kinesis's advanced stream processing capabilities.</p>

<img
  src="/images/posts/dynamodb-streams/7.webp"
  alt="Example workflow for delivering records from DynamoDB to Amazon S3 "
/>
<h4 style={{ marginTop: '5px' }}>Steps to Enable Kinesis Integration</h4>

<ul>
  <li>
    First, create a <strong>Kinesis Data Stream</strong> in the same region as
    your DynamoDB table if you don't have one already.
  </li>
  <li>
    Go to the <strong>DynamoDB console</strong> and select your table.
  </li>
  <li>
    Navigate to the <strong>Exports and streams</strong> tab.
  </li>
  <li>
    Scroll down to the <strong>Kinesis data stream details</strong> section.
  </li>
  <li>
    Click <strong>Enable Kinesis streaming destination</strong>.
  </li>
  <li>Select your Kinesis stream from the dropdown or enter its ARN.</li>
  <li>
    Click <strong>Enable</strong> to start streaming.
  </li>
</ul>

<p>
  Once enabled, DynamoDB will automatically replicate all item-level changes to
  the Kinesis stream. From there, we can process the data using{' '}
  <strong>Kinesis Data Analytics</strong>,{' '}
  <strong>Kinesis Data Firehose</strong> for delivery to S3 or Redshift, or
  multiple Lambda consumers for parallel processing.
</p>

<aside
  style={{
    borderLeft: '4px solid #2b6cb0',
    padding: '12px',
    background: '#f1f5f9',
    width: '100%',
  }}
>
  <strong>Note:</strong>
  <p style={{ margin: '8px 0 0' }}>
   This integration is especially useful when we need to process the same data changes 
   across multiple systems simultaneously, or when we want to leverage Kinesis's 
   built-in integrations with analytics and data warehousing services
  </p>
</aside>
</div>
</div>

<div style={{ marginTop: '20px' }}>
  <h3>Building a Custom Consumer</h3>
  <div style={{ padding: '5px' }}>
  <p>Now let's see how to build a <strong>custom consumer</strong> in Node.js using the AWS SDK.</p>

<p>
  Why build a custom consumer? Sometimes we need more control over how we
  process stream records, or we want to integrate with systems that don't have
  native AWS support. Building our own consumer gives us that flexibility.
</p>

<p>Let's see how this works in practice.</p>

```javascript
import { DynamoDBClient, UpdateTableCommand } from '@aws-sdk/client-dynamodb';
import { DescribeTableCommand } from '@aws-sdk/client-dynamodb';
import {
  DynamoDBStreamsClient,
  DescribeStreamCommand,
  GetShardIteratorCommand,
  GetRecordsCommand,
} from '@aws-sdk/client-dynamodb-streams';

const client = new DynamoDBClient({ region: 'eu-central-1' });
const streamsClient = new DynamoDBStreamsClient({ region: 'eu-central-1' });
```

<p>
  Sets up the AWS SDK clients needed to work with DynamoDB and DynamoDB Streams.
  We import the necessary commands for enabling streams, describing tables, and
  reading stream records. Then we create two clients: one for{' '}
  <strong>DynamoDB operations</strong> and another for{' '}
  <strong>DynamoDB Streams operations</strong>, both configured for the
  eu-central-1 region.
</p>

```javascript
async function consumeStream(streamArn) {
  console.log('Starting stream consumer...');

  const { StreamDescription } = await streamsClient.send(
    new DescribeStreamCommand({ StreamArn: streamArn })
  );

  console.log(`Found ${StreamDescription?.Shards?.length || 0} shard(s)`);

  for (const shard of StreamDescription?.Shards ?? []) {
    const shardId = shard.ShardId;
    console.log(`\nProcessing shard: ${shardId}`);

    const { ShardIterator } = await streamsClient.send(
      new GetShardIteratorCommand({
        StreamArn: streamArn,
        ShardId: shardId,
        ShardIteratorType: 'TRIM_HORIZON',
      })
    );

    let iterator = ShardIterator;

    while (iterator) {
      console.log('Polling for records...');

      const { Records, NextShardIterator } = await streamsClient.send(
        new GetRecordsCommand({ ShardIterator: iterator })
      );

      console.log(`Found ${Records?.length || 0} record(s)`);

      for (const record of Records ?? []) {
        handleRecord(record);
      }

      iterator = NextShardIterator;
      await new Promise((r) => setTimeout(r, 2000));
    }
  }

  console.log('Stream consumer finished');
}
```

<p>
  Reads from the DynamoDB Stream by looping through <strong>all shards</strong>{' '}
  and processing records from the oldest available (
  <strong>TRIM_HORIZON</strong>). It continuously polls for new records every 2
  seconds, processes each one through <strong>handleRecord</strong>, and keeps
  listening for changes to the table.
</p>

```javascript
function handleRecord(record) {
  console.log('Handle the record...');
}
```

<p>
  Defines the <strong>handleRecord</strong> function where we implement our
  custom logic to process each stream record. In this example, it simply logs a
  message, but in a real application, this is where we'd put any business logic
  we need for updating a cache, sending notifications, transforming data, or
  triggering other workflows.
</p>

```javascript
async function main() {
  const { Table } = await client.send(
    new DescribeTableCommand({ TableName: 'doodoot-table' })
  );
  console.log('Stream ARN:', Table?.LatestStreamArn);
  consumeStream(Table?.LatestStreamArn);
}

main();
```

<p>
  The <strong>main</strong> function brings everything together. It retrieves
  the table information to get the <strong>Stream ARN</strong>, logs it to the
  console, and then starts the consumer by calling{' '}
  <strong>consumeStream</strong> with that ARN. This kicks off the entire stream
  processing workflow.
</p>

<img src="/images/posts/dynamodb-streams/4.gif" alt="Testing" />

</div>
</div>

<div style={{ marginTop: '20px' }}>
  <h3>Conclusion</h3>
  <div style={{ padding: '5px' }}>
    <p>
      AWS DynamoDB Streams transforms NoSQL databases into{' '}
      <strong>event-driven powerhouses</strong>, capturing every insert, update,
      or delete as a real-time event. By integrating with{' '}
      <strong>Lambda functions</strong>, <strong>Kinesis Data Streams</strong>,
      or custom Node.js consumers, we can build reactive systems that power{' '}
      <strong>real-time analytics</strong>,{' '}
      <strong>cross-region replication</strong>,{' '}
      <strong>cache invalidation</strong>, and{' '}
      <strong>AI-driven workflows</strong>. Instead of treating DynamoDB as just
      a static datastore, Streams turns it into a{' '}
      <strong>live data pipeline</strong> that enables modern, event-driven
      architectures where services respond to changes without tight coupling or
      constant polling.
    </p>
  </div>
</div>


DynamoDB Streams


<div style={{ marginTop: '20px' }}>
  <h3>Introduction</h3>
  <div style={{ padding: '5px' }}>
    Amazon S3 is perfect for storing large volumes of unstructured data, but
    it’s not designed for rich querying. If your app needs to search files by
    extension, owner, date, or metadata, you’ll need an index. In this guide,
    we’ll build a production‑ready file indexing system.
  </div>
</div>

<div style={{ marginTop: '20px' }}>
  <h3>Why Index S3 Files</h3>
  <div style={{ padding: '5px' }}>
    S3 only supports basic prefix and pagination operations, making{' '}
    <strong>fast search</strong> nearly impossible for complex queries. When you
    need <strong>metadata queries</strong> to find files by tags, size, type, or
    owner, you're out of luck with S3 alone. That's where our{' '}
    <strong>scalable architecture</strong> comes in DynamoDB can handle billions
    of rows with <strong>single-digit millisecond latency</strong>, turning our
    file storage into a searchable powerhouse that actually responds when we
    need it to
  </div>
</div>

<div style={{ marginTop: '20px' }}>
  <h3>High-Level Architecture</h3>
  <div style={{ padding: '5px' }}>
    <p>
  During this post, we will build a <strong>fully serverless pipeline</strong> 
  to handle <strong>S3 file indexing</strong>. We will create an 
  <strong>automated system</strong> that tracks every 
  <strong> file upload</strong>, <strong>extraction</strong>, and 
  <strong> deletion</strong> in <strong>real-time</strong>, giving us 
  <strong> instant search capabilities</strong> across the entire 
  <strong> S3 bucket</strong>.
</p>

  </div>
</div>
<center>
  <img src="/images/posts/s3-file-indexer/1.png" />
</center>

  <div style={{ padding: '5px' }}>
<p>
  This architecture illustrates a <strong>serverless pipeline</strong> for managing <strong>S3 file events</strong> in real-time:
</p>

<p style={{ marginTop: '5px' }}>
  1. A <strong>user</strong> uploads or deletes a file in an{' '}
  <strong>S3 bucket</strong>. This action automatically triggers an{' '}
  <strong>S3 event</strong>.
</p>

<p style={{ marginTop: '4px' }}>
  2. The event is sent to an <strong>SQS Queue</strong>, which acts as a{' '}
  <strong>buffer</strong> to ensure that all events are processed reliably.
</p>

<p style={{ marginTop: '4px' }}>
  3. An <strong>AWS Lambda function</strong> pulls messages from the SQS queue.
  Lambda is <strong>serverless</strong>, meaning it automatically scales to
  handle the load and only runs when needed.
</p>

<p style={{ marginTop: '4px' }}>
  4. The Lambda function processes each message, updating the{' '}
  <strong>DynamoDB</strong> database to reflect the current state of files in
  the S3 bucket. This ensures that we always have an{' '}
  <strong>up-to-date index</strong> of all files.
</p>

<p style={{ marginTop: '5px' }}>
  This system provides a{' '}
  <strong>fully automated, real-time, and scalable solution</strong> for
  tracking file uploads, deletions, and modifications in an S3 bucket, all
  without managing any servers.
</p>
  </div>

<div style={{ marginTop: '20px' }}>
  <h3>Provision Resources</h3>
  <div style={{ padding: '5px' }}>
    <p>
      With our architecture in place, we are now ready to provision and
      implement all the necessary components starting by the DynamoDb table
    </p>
  </div>
</div>

<div style={{ marginTop: '20px' }}>
  <h4>DynamoDB Table</h4>
  <div style={{ padding: '5px' }}>
    <p>
     To set up our DynamoDB table, the first step is to determine which file Attributes we want to index. For this example, I have chosen the following Attributes
    </p>
  </div>
<div style={{textAlign:'center', marginTop : '5px'}}>
  <table border="1" cellpadding="8" cellspacing="0" style={{margin: '0 auto'}}>
    <thead>
      <tr>
        <th>Attribute</th>
        <th>Description</th>
      </tr>
    </thead>
    <tbody>
      <tr>
        <td>fileKey</td>
        <td>The full path or key in S3 (unique identifier)</td>
      </tr>
      <tr>
        <td>userId</td>
        <td>The unique identifier of the user who owns this file</td>
      </tr>
      <tr>
        <td>bucketName</td>
        <td>Name of the S3 bucket where the file is stored</td>
      </tr>
      <tr>
        <td>fileName</td>
        <td>File name (can be extracted from fileKey)</td>
      </tr>
      <tr>
        <td>fileSize</td>
        <td>Size in bytes</td>
      </tr>
      <tr>
        <td>fileType</td>
        <td>MIME type (e.g., application/pdf, image/png)</td>
      </tr>
      <tr>
        <td>extension</td>
        <td>File extension (e.g., .pdf, .jpg)</td>
      </tr>
      <tr>
        <td>uploadedAt</td>
        <td>Timestamp of when the file was uploaded</td>
      </tr>
    </tbody>
  </table>
</div>

<p>
  Now that we know what data to store, we are ready to create the table. For
  this example, I will create it manually, but you can also use CloudFormation,
  CDK or Terraform.
</p>

</div>

<center>
  <img src="/images/posts/s3-file-indexer/2.png" />
</center>

<aside
  style={{
    borderLeft: '4px solid #2b6cb0',
    padding: '12px',
    background: '#f1f5f9',
    width: '100%',
  }}
>
  <strong>Note:</strong>
  <p style={{ margin: '8px 0 0' }}>
    In the AWS Console you only define the table's primary key (partition key
    and optional sort key). All other attributes are schemaless and can be added
    later when inserting items.
  </p>
</aside>

<div style={{ marginTop: '20px' }}>
  <h4>S3 Bucket and SQS Queue</h4>
  <div style={{ padding: '5px' }}>
    <p>
      Now it’s time to provision the S3 bucket that will store our files, along
      with the SQS queue that will receive events from S3 whenever a file is
      uploaded or deleted.
    </p>

<p>
  Starting with the bucket, I created one called{' '}
  <strong>doodootiuploads</strong>, and for the queue, I created{' '}
  <strong>file-indexer-queue</strong>, as shown in the images below.
</p>

  </div>
</div>

<div style={{ textAlign: 'center', margin: '20px 0' }}>
  <img
    src="/images/posts/s3-file-indexer/3.png"
    alt="S3 bucket creation"
    style={{ maxWidth: '100%', borderRadius: '8px' }}
  />
</div>

<div style={{ textAlign: 'center', margin: '20px 0' }}>
  <img
    src="/images/posts/s3-file-indexer/4.png"
    alt="SQS queue creation"
    style={{ maxWidth: '100%', borderRadius: '8px' }}
  />
</div>

  <div style={{ padding: '5px' }}>
  <p>
  To allow S3 to send events to our SQS queue, we need to do some configuration.  
  The first step is to attach a <strong>queue policy</strong> to the SQS queue,  
  granting S3 permission to send events.
</p>

  </div>

```json
{
  "Version": "2012-10-17",
  "Id": "S3ToSQSPolicy",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "s3.amazonaws.com" },
      "Action": "SQS:SendMessage",
      "Resource": "arn:aws:sqs:us-east-1:123456789012:file-indexer-queue",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:s3:::doodootiuploads"
        }
      }
    }
  ]
}
```

<aside
  style={{
    borderLeft: '4px solid #2b6cb0',
    padding: '12px',
    background: '#f1f5f9',
    width: '100%',
    margin: '12px 0',
  }}
>
  <strong>Note:</strong>
  <p style={{ margin: '8px 0 0' }}>
    Remember to replace the SQS queue ARN and S3 bucket ARN with your own values
    when adding the queue policy for S3 to send events.
  </p>
</aside>

 <div style={{ padding: '5px' }}>
<p>
  Next, we need to create an <strong>event notification</strong> for our S3 bucket. 
  This ensures that whenever a file is uploaded or deleted, the SQS queue is triggered
   with the corresponding event.
</p>

<p>
  We create an event notification in our S3 bucket by going to its properties,
  selecting <strong>Event notifications</strong> and creating a new notification.
  We choose the events to track, like uploads or deletions, and set our SQS queue as the destination.
</p>
</div>

<div style={{ textAlign: 'center', margin: '20px 0' }}>
  <img
    src="/images/posts/s3-file-indexer/5.png"
    alt="Event notifications"
    style={{ maxWidth: '100%', borderRadius: '8px' }}
  />
</div>

 <div style={{ padding: '5px' }}>
<p>
  If you clicked <strong>Save changes</strong> and saw the error shown in the image below,  
  the issue is likely that the SQS queue access policy is not configured correctly.
</p>

</div>

<div style={{ textAlign: 'center', margin: '20px 0' }}>
  <img
    src="/images/posts/s3-file-indexer/6.png"
    alt="Event notifications"
    style={{ maxWidth: '100%', borderRadius: '8px' }}
  />
</div>

<div style={{ marginTop: '20px' }}>
  <h4>Lambda Function</h4>
  <div style={{ padding: '5px' }}>
    <p>
      The final and most important step is to create the Lambda function that
      will pull messages from the SQS queue, extract the metadata, and store
      this data in DynamoDB.
    </p>
  </div>
</div>

<aside
  style={{
    borderLeft: '4px solid #2b6cb0',
    padding: '12px',
    background: '#f1f5f9',
    width: '100%',
    margin: '12px 0',
  }}
>
  <strong>Note:</strong>
  <p style={{ margin: '8px 0 0' }}>
    I will use Node.js for the Lambda function code, but any runtime supported
    by AWS Lambda can be used.
  </p>
</aside>

<div style={{ textAlign: 'center', margin: '20px 0' }}>
  <img
    src="/images/posts/s3-file-indexer/7.png"
    alt="lambda function"
    style={{ maxWidth: '100%', borderRadius: '8px' }}
  />
</div>

<div style={{ padding: '5px' }}>
  <p>
  I started by creating a function called <strong>file-indexer-function</strong>.  
  To make this function work, we first need to configure it to allow reading messages from SQS,  
  and then add the appropriate code to process those messages.
</p>

<p>
  The first step is to attach an IAM policy to the Lambda execution role,
  granting it permission to receive messages from the SQS queue and to write
  data into the DynamoDB table.
</p>

</div>

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sqs:ReceiveMessage",
        "sqs:DeleteMessage",
        "sqs:GetQueueAttributes"
      ],
      "Resource": "arn:aws:sqs:us-east-1:123456789012:file-indexer-queue"
    },
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:PutItem",
        "dynamodb:UpdateItem",
        "dynamodb:DeleteItem"
      ],
      "Resource": "arn:aws:dynamodb:us-east-1:123456789012:table/S3-File-Indexer"
    }
  ]
}
```

<aside
  style={{
    borderLeft: '4px solid #2b6cb0',
    padding: '12px',
    background: '#f1f5f9',
    width: '100%',
    margin: '12px 0',
  }}
>
  <strong>Note:</strong>
  <p style={{ margin: '8px 0 0' }}>
    To give your Lambda the proper permissions, go to the function's
    <strong>Configuration → Permissions</strong> tab, click the execution role, and
    attach a policy that allows access to both services:
    <strong>sqs:ReceiveMessage</strong>, <strong>sqs:DeleteMessage</strong>, and{' '}
    <strong>sqs:GetQueueAttributes</strong> for your SQS queue, as well as
    <strong>dynamodb:PutItem</strong>, <strong>dynamodb:UpdateItem</strong>, and{' '}
    <strong>dynamodb:DeleteItem</strong> for your DynamoDB table.
  </p>
</aside>

<div style={{ padding: '5px' }}>
<p>
  Next, from the Lambda UI, we need to add the SQS queue as a trigger for our function.
</p>

</div>

<div style={{ textAlign: 'center', margin: '20px 0' }}>
  <img
    src="/images/posts/s3-file-indexer/8.png"
    alt="lambda function"
    style={{ maxWidth: '100%', borderRadius: '8px' }}
  />
</div>

<div style={{ padding: '5px' }}>
  <p>
    With all the configuration complete, we just need to create the code logic
    that will process these events and save the data into DynamoDB and to make
    the code simple and easy to follow, I will split it into steps so you can
    follow along.
  </p>
</div>

<div style={{ padding: '5px' }}>
  <h5>Step 1: Import AWS SDK and create DynamoDB client</h5>
</div>

```js
import { S3Client, HeadObjectCommand } from '@aws-sdk/client-s3';
import {
  DynamoDBClient,
  PutItemCommand,
  DeleteItemCommand,
} from '@aws-sdk/client-dynamodb';

const s3Client = new S3Client({ region: 'eu-central-1' });
const dynamoClient = new DynamoDBClient({ region: 'eu-central-1' });
const TABLE_NAME = 'S3-File-Indexer';
```

<p style={{ padding: '5px' }}>
  We import clients for <strong>S3</strong> and <strong>DynamoDB</strong>,
  create client instances to interact with them.
</p>

<aside
  style={{
    borderLeft: '4px solid #2b6cb0',
    padding: '12px',
    background: '#f1f5f9',
    width: '100%',
    margin: '12px 0',
  }}
>
  <strong>Note:</strong>
  <p style={{ margin: '8px 0 0' }}>
    We can use environment variables for the region and table name, but for the
    sake of this post, I have used hard-coded values.
  </p>
</aside>

<div style={{ padding: '5px' }}>
  <h5>Step 2: Lambda handler and iterating over SQS records</h5>
</div>

<p>Entry Point & Iterating SQS Messages</p>

```js
export const handler = async (event) => {
  console.log("Processing SQS messages with S3 events");

  for (const sqsRecord of event.Records) {
    try {
      // Parse the S3 event from the SQS message body
      const s3Event = JSON.parse(sqsRecord.body);
```

<p>Extracting File Info from S3 Event</p>

```js
// Process each S3 record within the SQS message
for (const s3Record of s3Event.Records) {
  const bucketName = s3Record.s3.bucket.name;
  const fileKey = decodeURIComponent(s3Record.s3.object.key.replace(/\+/g, ' '));
  const fileSize = s3Record.s3.object.size;
  const eventName = s3Record.eventName;

if (eventName.startsWith("ObjectCreated")) {
```

<p>Fetching Metadata from S3</p>

```js
// Get metadata from the S3 object
let metadata = {};
let contentType = 'unknown';

try {
  const headResult = await s3Client.send(
    new HeadObjectCommand({
      Bucket: bucketName,
      Key: fileKey,
    })
  );
  metadata = headResult.Metadata || {};
  contentType = headResult.ContentType || 'unknown';
} catch (err) {
  console.error(`Error fetching S3 metadata for ${fileKey}:`, err.message);
}
```

<p>Building File Details</p>

```js
const fileName = fileKey.split('/').pop() || fileKey;
const extension = fileName.includes('.') ? '.' + fileName.split('.').pop() : '';
const fileType = metadata['content-type'] || contentType;
const uploadedAt =
  metadata['upload-timestamp'] || new Date(s3Record.eventTime).toISOString();
const userId = metadata['user-id'] || 'unknown';
```

<p style={{ padding: '5px' }}>
  We fetch object metadata from S3 using <strong>HeadObjectCommand</strong>.
  Metadata includes <strong>user ID</strong>, upload timestamp, and optionally
  file type. Fallbacks are used if metadata is missing.
</p>

<div style={{ padding: '5px' }}>
  <h5>Step 3: Storing uploaded file metadata in DynamoDB</h5>
</div>

```js
// Store metadata in DynamoDB
await dynamoClient.send(
  new PutItemCommand({
    TableName: TABLE_NAME,
    Item: {
      bucketName: { S: bucketName },
      fileKey: { S: fileKey },
      fileName: { S: fileName },
      extension: { S: extension },
      fileType: { S: fileType },
      fileSize: { N: fileSize.toString() },
      uploadedAt: { S: uploadedAt },
      userId: { S: userId },
      createdAt: { S: new Date().toISOString() },
      etag: { S: s3Record.s3.object.eTag || '' },
    },
  })
);

console.log(`File ${fileKey} added to DynamoDB`);
}
```

<p style={{ padding: '5px' }}>
  Metadata is stored in DynamoDB using <strong>PutItemCommand</strong>. Each
  attribute is typed (S for string, N for number), and logs confirm successful
  insertion.
</p>

<div style={{ padding: '5px' }}>
  <h5>Step 4: Handling file deletions (ObjectRemoved)</h5>
</div>

```js
 // Delete from DynamoDB
 else if (eventName.startsWith("ObjectRemoved")) {
     await dynamoClient.send(new DeleteItemCommand({
            TableName: TABLE_NAME,
            Key: {
              bucketName: { S: bucketName },
              fileKey: { S: fileKey },
            },
          }));

          console.log(`File ${fileKey} deleted from DynamoDB`);
        }
      }
    } catch (error) {
      console.error("Error processing SQS message:", error.message);
      throw error; // Re-throw to trigger Lambda retry
    }
  }

  return {
    statusCode: 200,
    body: JSON.stringify({ message: "Successfully processed S3 events" })
  };
};
```

<p style={{ padding: '5px' }}>
  For deletions, the corresponding item is removed from DynamoDB using{' '}
  <strong>DeleteItemCommand</strong>. Only the primary key attributes (
  <strong>bucketName</strong> and <strong>fileKey</strong>) are required.
</p>

<div style={{ padding: '5px' }}>
  <h5>Step 5: Testing</h5>
  <p>
    Now that the code is complete, it’s time to test it. I’ll use the S3 console
    to upload a file, and as shown in the image below, the file was indexed
    successfully.
  </p>
</div>

<div style={{ textAlign: 'center', margin: '20px 0' }}>
  <img
    src="/images/posts/s3-file-indexer/9.png"
    alt="lambda function"
    style={{ maxWidth: '100%', borderRadius: '8px' }}
  />
</div>

<aside
  style={{
    borderLeft: '4px solid #2b6cb0',
    padding: '12px',
    background: '#f1f5f9',
    width: '100%',
    margin: '12px 0',
  }}
>
  <strong>Note:</strong>
  <p>
  To include the user ID in the metadata, we can add this logic when uploading the file using the S3 client, as shown in the code below.
</p>

</aside>

```js
import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3';

const s3 = new S3Client({ region: 'eu-central-1' });

// Simple upload with userId
const uploadFile = async (file, userId) => {
  await s3.send(
    new PutObjectCommand({
      Bucket: 'doodootiuploads',
      Key: `users/${userId}/${file.name}`,
      Body: file,
      ContentType: file.type,
      Metadata: {
        'user-id': userId,
        'upload-timestamp': new Date().toISOString(),
      },
    })
  );
};
```

<div style={{ marginTop: '20px' }}>
  <h3>Conclusion</h3>
  <div style={{ padding: '5px' }}>
  <p>
  We’ve built a fully serverless pipeline that indexes files uploaded to S3 by capturing events, processing them with Lambda, and storing metadata in DynamoDB.  
  This approach is scalable, cost-effective, and can be extended to support more complex workflows as needed.
</p>

  </div>
</div>


Serverless S3 File Indexing Made Simple


<div style={{ marginTop: "20px" }}>
  <h3>Introduction</h3>
  <div style={{ padding: "5px" }}>
    <p>
      <strong>AWS Lambda</strong> is at the heart of building serverless applications on AWS. It lets us run our code without worrying about servers, but the way we <strong>trigger</strong> our Lambda functions, and how we connect them with other AWS services, can have a big impact on our system's performance and scalability. 
      <br /><br />
      In this post, we'll explore the different <strong>Lambda invocation types</strong>, dive into <strong>event source mappings</strong>, and walk through advanced usage patterns with <strong>Node.js</strong>. We'll also see practical examples, learn best practices, and pick up tips for deploying in the real world.
    </p>
  </div>
</div>


<div style={{ marginTop: "20px" }}>
  <h3>Lambda Invocation Types</h3>
  <div style={{ padding: "5px" }}>
    <p>
      There are two main ways to invoke a Lambda function:
      <ul>
        <li><strong>Synchronous</strong> (RequestResponse): The caller waits for the function to finish and gets the result immediately. Used by API Gateway, SDK calls, etc.</li>
        <li><strong>Asynchronous</strong> (Event): The caller hands off the event and moves on. Lambda processes it later. Used by S3, SNS, EventBridge, etc.</li>
      </ul>
    </p>
    <p>Let's see both in action with Node.js:</p>
  </div>
</div>

 <div style={{ padding: "5px" }}>
    <p>
     In the following example, we demonstrate how to invoke a Lambda function synchronously from another Lambda function. With synchronous invocation, the caller waits for the function to complete execution and receive the response before continuing.
    </p>
  </div>

```js
// Synchronous invocation (RequestResponse)
const AWS = require('aws-sdk');
const lambda = new AWS.Lambda();

exports.handler = async (event) => {
  const params = {
    FunctionName: 'name-of-the-other-lambda-function',
    Payload: JSON.stringify({ key: 'value' }),
    InvocationType: 'RequestResponse',
  };
  const response = await lambda.invoke(params).promise();
  return JSON.parse(response.Payload);
};
```
 <div style={{ padding: "5px" }}>
    <p>
     In the following example, we invoke a Lambda function asynchronously using the 'Event' invocation type. 
     With asynchronous invocation, we don't wait for or expect a response cuz Lambda processes the event in the background and returns immediately.
    </p>
  </div>


```js
// Asynchronous invocation (Event)
const params = {
  FunctionName: 'lambda-name',
  Payload: JSON.stringify({ key: 'value' }),
  InvocationType: 'Event',
};
await lambda.invoke(params).promise();
```

<div style={{ marginTop: "20px" }}>
  <h3>Event Source Mapping</h3>
  <div style={{ padding: "5px" }}>
    <p>
      <strong>Event Source Mapping</strong> is how Lambda connects to event sources like <strong>SQS</strong>, <strong>DynamoDB Streams</strong>, and <strong>Kinesis</strong>. AWS automatically polls these sources and invokes our function with batches of events. This is the secret sauce for building <strong>event-driven</strong> architectures
    </p>
    <center>
      <img src="/images/posts/lambda-invocations/1.png" style={{ marginTop: "5px" }} />
      <small>Example of DynamoDB Streams</small>
    </center>
    <p style={{ marginTop: "10px" }}>
      We can create event source mappings via the AWS Console, CLI, or programmatically:
    </p>
  </div>
</div>

```js
// Create an SQS event source mapping for a Lambda function
const AWS = require('aws-sdk');
const lambda = new AWS.Lambda();

async function createEventSourceMapping() {
  const params = {
    EventSourceArn: 'arn:aws:sqs:REGION:ACCOUNT_ID:queue-name',
    FunctionName: 'lambda-name',
    BatchSize: 10,
    Enabled: true,
  };
  const result = await lambda.createEventSourceMapping(params).promise();
  console.log('Mapping created:', result.UUID);
}
```

<div style={{ padding: "5px" }}>
  <h4>Handling Batched Events in Node.js</h4>
  <p>When Lambda is triggered by SQS, DynamoDB Streams, or Kinesis, it receives a batch of records:</p>
</div>

```js
exports.handler = async (event) => {
  for (const record of event.Records) {
    // For SQS: record.body
    // For DynamoDB Streams: record.dynamodb
    // For Kinesis: Buffer.from(record.kinesis.data, 'base64').toString('utf-8')
    console.log('Processing record:', record);
  }
};
```

<div style={{ padding: "5px" }}>
  <h4>Advanced: Partial Batch Response (SQS)</h4>
  <p>
    With SQS, we can avoid reprocessing successful messages in a failed batch by returning a <strong>partial batch response</strong>:
  </p>
</div>

```js
exports.handler = async (event) => {
  const batchItemFailures = [];
  for (const record of event.Records) {
    try {
      // Process record
    } catch (err) {
      batchItemFailures.push({ itemIdentifier: record.messageId });
    }
  }
  return { batchItemFailures };
};
```

<div style={{ marginTop: "20px" }}>
  <h3>Advanced Invocation Patterns</h3>
  <div style={{ padding: "5px" }}>
    <p>
      <strong>AWS Lambda</strong> offers several sophisticated patterns for building complex <strong>serverless applications</strong>. You can create <strong>modular workflows</strong> by <strong>chaining Lambda functions</strong> together, where one function directly invokes another to create a sequence of operations. For parallel processing needs, <strong>EventBridge</strong> enables <strong>fan-out patterns</strong> where a single event can trigger multiple <strong>Lambda functions</strong> simultaneously, allowing for efficient <strong>distributed processing</strong>.
    </p>
    <p>
      When you need more control over your function workflows, <strong>AWS Step Functions</strong> provides powerful <strong>orchestration capabilities</strong>. It allows you to coordinate multiple <strong>Lambda functions</strong> with built-in <strong>error handling</strong> and <strong>automatic retries</strong>, making it ideal for complex business processes. For improved reliability, you can implement <strong>error handling strategies</strong> using <strong>Dead Letter Queues (DLQs)</strong> and <strong>custom retry logic</strong>, ensuring your applications remain resilient even when issues occur.
    </p>
    <p>
      Additionally, Lambda supports <strong>custom invocation contexts</strong>, allowing you to pass <strong>metadata</strong> for various purposes such as <strong>distributed tracing</strong>, <strong>feature flags</strong>, and other application-specific needs. This flexibility enables you to build sophisticated <strong>monitoring</strong> and <strong>control mechanisms</strong> into your <strong>serverless applications</strong>.
    </p>
  </div>
</div>

<div style={{ padding: "5px" }}>
  <h4>Chaining Lambdas Example</h4>
  <p>
    Here's an example of chaining Lambda functions, where one function invokes another. 
    This pattern is useful when you need to break down complex operations into smaller, more manageable steps.
     The first Lambda function can process data and then pass the results to a second function for further processing. 
     This creates a sequential workflow where functions execute one after another.
  </p>
</div>

```js
// Lambda A invokes Lambda B
await lambda.invoke({
  FunctionName: 'LambdaB',
  Payload: JSON.stringify({ foo: 'bar' }),
  InvocationType: 'Event',
}).promise();
```

<div style={{ padding: "5px" }}>
  <h4>Fan-out with EventBridge</h4>
  <p>
    EventBridge enables powerful fan-out patterns where a single event can trigger multiple Lambda functions simultaneously.
     This is particularly useful for scenarios where you need parallel processing or want to decouple different parts of your application. 
     For example, when a new user signs up, you might want to simultaneously send a welcome email, create a user profile, 
     and trigger analytics tracking  all as separate Lambda functions processing the same event.
      The following example shows how to publish an event that multiple Lambda functions can subscribe to and process independently.
  </p>
</div>

```js
const eventBridge = new AWS.EventBridge();
await eventBridge.putEvents({
  Entries: [{
    Source: 'my.app',
    DetailType: 'task',
    Detail: JSON.stringify({ taskId: 123 }),
    EventBusName: 'default',
  }]
}).promise();
```

<div style={{ padding: "5px" }}>
  <h4>Step Functions for Orchestration</h4>
  <p>We can use Step Functions to coordinate multiple Lambdas with error handling and retries.</p>
</div>

```json
{
  "StartAt": "FirstLambda",
  "States": {
    "FirstLambda": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:FirstLambda",
      "Next": "SecondLambda"
    },
    "SecondLambda": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:SecondLambda",
      "End": true
    }
  }
}
```

<div style={{ padding: "5px" }}>
  <h4>Error Handling and DLQs</h4>
  <p>We can configure Dead Letter Queues (DLQs) and custom retry logic for failed events.</p>
</div>

```js
try {
  // Lambda logic
} catch (err) {
  // Send to SQS DLQ or log error
}
```

<div style={{ marginTop: "20px" }}>
  <h3>Node.js Lambda Invocation Patterns</h3>
  <div style={{ padding: "5px" }}>
    <p>When working with <strong>AWS Lambda functions</strong> in <strong>Node.js</strong>, there are several key <strong>invocation patterns</strong> we need to understand.
     First, we can programmatically invoke <strong>Lambda functions</strong> using the <strong>AWS SDK</strong>, which gives us fine-grained control over function execution.
      The <strong>SDK</strong> allows us to specify <strong>invocation types</strong> and pass custom <strong>payloads</strong> to our functions. We also have access to the powerful <strong>context object</strong>,
       which provides important <strong>metadata</strong> and <strong>utilities</strong> for handling the function's execution environment. 
       This includes things like the <strong>request ID</strong>, remaining <strong>execution time</strong>, and <strong>logging capabilities</strong>.
        Additionally, we need robust <strong>error handling patterns</strong> to gracefully handle failures and ensure our functions are resilient. 
        Finally, Lambda supports <strong>parallel invocations</strong>, enabling us to execute multiple functions concurrently for better <strong>performance</strong> and <strong>scalability</strong> in 
        distributed architectures.</p>
  </div>
</div>

```js
// Programmatic invocation utility
async function invokeLambda(functionName, payload, type = 'RequestResponse') {
  const params = {
    FunctionName: functionName,
    Payload: JSON.stringify(payload),
    InvocationType: type,
  };
  const result = await lambda.invoke(params).promise();
  return result.Payload ? JSON.parse(result.Payload) : null;
}
```

```js
// Handling invocation context
exports.handler = async (event, context) => {
  console.log('Request ID:', context.awsRequestId);
  // We can use context for tracing, timeouts, etc.
};
```

```js
// Error handling pattern
exports.handler = async (event) => {
  try {
    // Our logic
  } catch (error) {
    console.error('Error:', error);
    throw new Error('Custom error message');
  }
};
```

```js
// Parallel invocations (fan-out)
const functions = ['LambdaA', 'LambdaB', 'LambdaC'];
await Promise.all(functions.map(fn =>
  lambda.invoke({
    FunctionName: fn,
    Payload: JSON.stringify({ key: 'value' }),
    InvocationType: 'Event',
  }).promise()
));
```

<div style={{ marginTop: "20px" }}>
  <h3>Best Practices</h3>
<div style={{ padding: "5px" }}>
  <p>
    To follow best practices with AWS Lambda, make sure to use <strong>environment variables</strong> for configuration settings. For <strong>event-driven Lambdas</strong>, implement <strong>idempotency</strong> to avoid processing the same event more than once. Set up monitoring with <strong>CloudWatch Logs</strong> and <strong>X-Ray</strong> to track performance and troubleshoot issues.
    <br /><br />
    Always secure your functions using <strong>least privilege IAM roles</strong>. For handling errors in asynchronous invocations, use <strong>Dead Letter Queues (DLQs)</strong>. When working with event source mappings, fine-tune <strong>batch size</strong> and <strong>concurrency</strong> settings for optimal performance. If you're using <strong>SQS</strong> as a trigger, enable <strong>partial batch responses</strong> to prevent message loss or duplication.
  </p>
</div>

</div>

<div style={{ marginTop: "20px" }}>
  <h3>Conclusion</h3>
  <div style={{ padding: "5px" }}>
    <p>
      Understanding how Lambda invocation types and event source mappings work is key to getting the most out of AWS serverless. With <strong>Node.js</strong>, we have the tools to build systems that are scalable, responsive, and easy to maintain. 
 </p>
  </div>
</div>


Advanced AWS Lambda Invocations with Node.js


<div style={{ marginTop: "20px" }}>
  <h3>Introduction</h3>
  <div style={{ padding: "5px" }}>
     <strong>AWS Lambda</strong> is a powerful serverless compute service, but as your functions grow, managing <strong>dependencies</strong> and <strong>code reuse</strong> can become challenging.
    <strong>Lambda Layers</strong> are AWS's answer to this problem, allowing you to package <strong>libraries</strong>, <strong>custom runtimes</strong>, 
    and other <strong>dependencies</strong> separately from your function code. In this post, we'll explore <strong>Lambda Layers</strong> in depth, walk through <strong>Node.js</strong> examples,
    and review the main ways to deploy <strong>Lambda functions</strong>.
  </div>
</div>

<div style={{ marginTop: "20px" }}>
  <h3>What are AWS Lambda Layers?</h3>
  <center>
  <img src="/images/posts/lambda-layers/1.png" />
</center>
  <div style={{ padding: "5px" }}>
    <p>
      <strong>Lambda Layers</strong> are <strong>ZIP archives</strong> containing <strong>libraries</strong>, <strong>custom runtimes</strong>, or other <strong>dependencies</strong>.
      You can attach up to 5 layers to a <strong>Lambda function</strong>, and <strong>AWS</strong> merges them with your function code at runtime.
    </p>
   <p> This separation allows for:</p>
    <ul>
      <li>- <strong>Code reuse</strong> across multiple functions</li>
      <li>- <strong>Smaller deployment packages</strong></li>
      <li>- <strong>Easier dependency management</strong></li>
    </ul>
  </div>
</div>

<div style={{ marginTop: "20px" }}>
  <h3>Creating a Lambda Layer</h3>
<div style={{ padding: "5px" }}>
    As always, we will take an <strong>example</strong> and <strong>implement</strong> it in order to understand how <strong>Lambda function layers</strong> work. Let's say that we have a <strong>library</strong> called <strong>Lodash</strong> (loadash.js) that we want 
    to use in many <strong>functions</strong>. In the normal case without using <strong>layers</strong>, we will <strong>package</strong> this <strong>library</strong> with each <strong>function</strong> and <strong>deploy</strong> it to <strong>AWS</strong> and this is not <strong>efficient</strong>.
    That is why we need to use <strong>Layers</strong> to make this <strong>library</strong> accessible across <strong>functions</strong>.
  </div>
    <h4>Prepare the Layer Content</h4>
</div>

 ```bash 
     mkdir -p lambda-layer/nodejs
     cd lambda-layer/nodejs
     npm init -y
     npm install lodash
 ```

 
<div style={{ marginTop: "20px" }}>
    <h4>Package the Layer</h4>
  <div style={{ padding: "5px" }}>
  <p>In order to <strong>publish</strong> our <strong>layer</strong> to <strong>production</strong> and make it <strong>shareable</strong> across <strong>functions</strong>, we first need to <strong>package</strong> it into a <strong>ZIP file</strong>.
   The <strong>ZIP file structure</strong> is important for <strong>Node.js layers</strong>, <strong>dependencies</strong> must be placed in a '<strong>nodejs/node_modules</strong>' directory.</p>
  </div>
</div>
 <center>
  <img src="/images/posts/lambda-layers/3.png" />
</center>

 ```bash 
   cd ..

   //zip 
   zip -r lodash-layer.zip nodejs

   //tar
   tar -a -cf lodash-layer.zip nodejs
 ```
 <center>
  <img src="/images/posts/lambda-layers/2.png" />
</center>

<div style={{ marginTop: "20px" }}>
    <h4>Publish the Layer</h4>
  <div style={{ padding: "5px" }}>
  <p>Now that our <strong>ZIP file</strong> is ready, we can <strong>publish</strong> our <strong>Layer</strong> using the <strong>AWS CLI</strong>. 
  The <strong>AWS CLI</strong> provides <strong>commands</strong> to <strong>create</strong> and <strong>manage</strong> <strong>Lambda layers</strong> directly from the <strong>terminal</strong>.
   This allows us to <strong>automate</strong> the layer <strong>publishing process</strong> and <strong>integrate</strong> it into our <strong>deployment workflows</strong>.</p>
  </div>
</div>

 ```bash 
 aws lambda publish-layer-version \
  --layer-name lodash-layer \
  --description "Lodash for Node.js Lambda" \
  --zip-file fileb://lodash-layer.zip \
  --compatible-runtimes nodejs18.x nodejs20.x
 ```


<div style={{ marginTop: "2px" }}>
  <div style={{ padding: "5px" }}>
    <p>After executing the command, we can see that it returns important information about the layer, including the ARN (Amazon Resource Name) which we'll need 
    to reference the layer later.</p>
 
  <p  style={{ marginTop: "7px" }}>Now if we check the AWS Console, we'll find that our lodash-layer has been successfully uploaded. While it's possible to publish layers directly through the AWS Console UI, I prefer using the CLI for better automation and version control capabilities.</p>
  </div>
</div>

  <center>
  <img src="/images/posts/lambda-layers/05.png" />
</center>

<div style={{ marginTop: "20px" }}>
    <h4> Use the Layer in a Lambda Function</h4>
  <div style={{ padding: "5px" }}>
  <p>Now that we have our layer published, we need to attach it to our Lambda function. We can do this in two ways: either through the AWS Console's UI,
   or programmatically using the AWS CLI and I'll show you the CLI command in just a moment but first we need to create a lambda function and deploy it in order 
   to test our layer.</p>
  </div>
</div>

 ```bash 
   mkdir doodooti-fn 
   cd doodooti-fn
   touch index.js
 ```

 ```js
 // index.js

 // this will be imported from the layer
 const _ = require('lodash');

exports.handler = async (event) => {
  // Use lodash as usual
  return _.capitalize('hello from lambda layer!');
};
 ```

  <div style={{ padding: "5px" }}>
  <p>Now that our function is ready, we can deploy it and link it to the Lodash layer. To do this, we first need to zip the function and use the AWS CLI to publish it. Don't forget to create an IAM execution role first, as it's mandatory for deploying Lambda functions.</p>
  </div>

```bash
  zip doodooti-fn.zip index.js

  aws lambda create-function \
  --function-name doodooti-fn \
  --runtime nodejs20.x \
  --role arn:aws:iam::ACCOUNT_ID:role/EXECUTION_ROLE \
  --handler index.handler \
  --zip-file fileb://doodooti-fn.zip \
  --layers arn:aws:lambda:us-east-1:ACCOUNT_ID:layer:lodash-layer:1
```

  <div style={{ padding: "5px" }}>
  <p>After executing the deployment command and receiving a success response, we can go to the AWS Console and verify that the function was deployed successfully.</p>
  </div>

 <center>
  <img src="/images/posts/lambda-layers/7.png" />
</center>

  <div style={{ padding: "5px" }}>
  <p>When we click into it, we can see that it has one layer attached and we can also view the function code.</p>
  </div>

   <center>
  <img src="/images/posts/lambda-layers/8.png" />
</center>

  <div style={{ padding: "5px" }}>
  <p>We can also go to the Test tab and test our function's behavior by creating a test event with any input payload. 
  Since our function doesn't use the event parameter, we can leave it empty.
   After running the test, we should see the capitalized string "Hello from lambda layer!" in the execution results, confirming that our Lambda function is working correctly with the Lodash layer.</p>
  </div>

 <center>
  <img src="/images/posts/lambda-layers/9.png" />
</center>

<div style={{ marginTop: "20px" }}>
    <h3>Sharing Custom Code</h3>
  <div style={{ padding: "5px" }}>
  <p>Besides sharing <strong>dependencies</strong>, <strong>Lambda Layers</strong> also allow us to share <strong>utility code</strong> that can be used across <strong>multiple functions</strong>.</p>
  </div>
</div>

```js
// lambda-layer/nodejs/utils.js
module.exports.hello = () => 'Hello from shared utils!';
```

```js
// will be imported from the layer
const utils = require('utils');

exports.handler = async (event) => {
  return utils.hello();
};
```

<div style={{ marginTop: "20px" }}>
    <h3>Custom runtimes</h3>
  <div style={{ padding: "5px" }}>
  <p>Lambda Layers also enable us to create custom runtimes for languages that AWS Lambda doesn't natively support.
   By packaging the necessary <strong>runtime components</strong>, such as <strong>interpreters</strong> and <strong>language-specific libraries</strong>, into a <strong>layer</strong>,
    we can extend Lambda's capabilities beyond its <strong>built-in runtimes</strong>. 
    This approach gives us the flexibility to run functions in our preferred <strong>programming language</strong> while maintaining the <strong>serverless benefits</strong> of AWS Lambda.
    While we won't deep dive into <strong>custom runtimes</strong> as they require complex implementation details, some examples include <strong>Rust</strong>, <strong>C++</strong> runtimes that the community has created.</p>
  </div>
</div>

<div style={{ marginTop: "20px" }}>
  <h3>Conclusion</h3>
  <div style={{ padding: "5px" }}>
    <strong>AWS Lambda Layers</strong> provide a powerful way to manage and share code across multiple <strong>Lambda functions</strong>.
     By separating <strong>dependencies</strong> and <strong>shared utilities</strong> into <strong>layers</strong>, we can reduce <strong>duplication</strong>, simplify <strong>maintenance</strong>, 
     and keep our Lambda functions focused on their core <strong>business logic</strong>. Whether you're sharing <strong>third-party dependencies</strong>, <strong>custom utility code</strong>,
      or implementing <strong>custom runtimes</strong>, <strong>Lambda Layers</strong> help you build more <strong>maintainable</strong> and <strong>efficient serverless applications</strong>.
       As your <strong>serverless architecture</strong> grows, leveraging <strong>Lambda Layers</strong> becomes increasingly valuable for managing <strong>complexity</strong> 
       and promoting <strong>code reuse</strong> across your functions.
  </div>
</div>









Lambda Layers Deep Dive & Deployment Strategies


<div style={{ marginTop: '20px' }}>
  <h3>Introduction</h3>
  <div style={{ padding: '5px' }}>
    <p>
      <strong>JavaScript's event loop</strong> is what makes Node.js's{' '}
      <strong>non-blocking operations</strong> work, but understanding how it
      handles tasks can be tricky! 🧩 In this guide, we’ll explain the
      difference between <strong>microtasks</strong> and{' '}
      <strong>macrotasks</strong> 🔍 and how they affect your Node.js apps.
      🚀{' '}
    </p>
  </div>
</div>

<div style={{ marginTop: '20px' }}>
  <h3>How Event Loop works</h3>
  <div style={{ padding: '5px' }}>
  <p>
  The event loop is a fundamental part of JavaScript's non-blocking model, especially in environments like Node.js. It allows JavaScript to handle asynchronous operations like I/O, timers, and events without blocking the main thread.
  </p>
    <p style={{ marginTop: '5px' }}>
  During the execution of <strong>synchronous code</strong>, any <strong>asynchronous operation</strong> will be placed into a specific queue based on its type to be processed later, after the synchronous code has been executed. 
  If it's a <strong>macrotask</strong>, it will be placed in the <strong>macrotask queue</strong> if it's a <strong>microtask</strong>, it will go into the <strong>microtask queue</strong>.
  The <strong>event loop</strong> handles these asynchronous operations by checking the queues. It always checks the <strong>microtask queue</strong> first because it has <strong>higher priority</strong> than the <strong>macrotask queue</strong>. 
  This ensures that small tasks are handled quickly before larger ones.
</p>

<h5 style={{ marginTop: '5px' }}> Iterations or Cycles </h5>
<p>
  An iteration (or cycle) occurs when the event loop picks a task from one of
  the queues and starts processing it. While the task is being executed, we
  refer to this period as the current iteration 
</p>
  </div>
</div>

<div style={{ marginTop: '20px' }}>
  <h3>What Are Tasks?</h3>
  <div style={{ padding: '5px' }}>
    <p>
      Before we get into the details, let’s talk about <strong>tasks</strong> in
      Node.js. Tasks are bits of work that the JavaScript engine needs to
      handle—like running code, responding to events, or executing callbacks. In
      Node.js, tasks are categorized into <strong>microtasks</strong> and{' '}
      <strong>macrotasks</strong> . How these tasks are scheduled and run can
      make a big difference in how your app performs and behaves.
    </p>
  </div>
</div>

<div style={{ marginTop: '20px' }}>
  <h3>Macrotasks (Tasks)</h3>
  <div style={{ padding: '5px' }}>
    <p>
      Macrotasks are what we typically call <strong>"tasks"</strong> in Node.js.
      They represent the broader units of work that need to be completed,
      executed one at a time in the <strong>event loop</strong>. Here are some
      common examples of macrotasks, with explanations and examples:
    </p>
  </div>
</div>

<div style={{ padding: '5px' }}>
  <h5>⏳ setTimeout()</h5>
  <p>Schedules a task to run after a specified delay.</p>
</div>

```js
setTimeout(() => {
  console.log('This is will run after 10ms');
}, 10);
```

<div style={{ padding: '5px' }}>
  <h5> setInterval()</h5>
  <p>Runs a task repeatedly at specified intervals</p>
</div>

```js
setInterval(() => console.log('Repeating every 2 seconds!'), 2000);
```

<div style={{ padding: '5px' }}>
  <h5>setImmediate()</h5>
  <p>Executes a task immediately after the current event loop iteration</p>
</div>

```js
setImmediate(() =>
  console.log('Executed immediately after the current iteration.')
);
```

<div style={{ padding: '5px' }}>
  <h5>I/O operations</h5>
  <p>Handles tasks like file reading/writing or network requests</p>
</div>

```js
fs.readFile('file.txt', () => console.log('File read complete!'));
```

<div style={{ padding: '5px' }}>
  <h5>Event callbacks</h5>
  <p>
    Executes a task in response to an event, like a button click or an incoming
    request
  </p>
</div>

```js
button.addEventListener('click', () => console.log('Button clicked!'));
```

<div style={{ padding: '5px' }}>
  <h5>UI rendering</h5>
  <p> Handles rendering tasks in browser-based environments</p>
</div>

```js
//Rendering a DOM change after user interaction
```

<div style={{ marginTop: '20px' }}>
  <h3>Microtasks</h3>
  <div style={{ padding: '5px' }}>
    <p>
      {' '}
      <strong>Microtasks</strong> are smaller units of work that are executed immediately
      after the current synchronous code completes, but before the next <strong>
        macrotask
      </strong> begins. ⏳ They have <strong>higher priority</strong> than macrotasks.
      Common sources of microtasks include:
    </p>
  </div>
</div>

<div style={{ padding: '5px' }}>
  <h5>Promise callbacks</h5>
  <p>
    These are used to handle the result or error of a <strong>Promise</strong>{' '}
    after it resolves or rejects. They are added to the{' '}
    <strong>microtask queue</strong> to be executed after the current
    synchronous code finishes, but before any macrotasks.
  </p>
</div>

```js
Promise.resolve().then(() => {
  console.log('This is a microtask');
});
```

<div style={{ padding: '5px' }}>
  <h5> process.nextTick()</h5>
  <p>
    This function schedules a callback to be invoked in the next iteration of
    the event loop, before any other I/O or timers. It has the highest priority
    among all microtasks.
  </p>
</div>

```js
process.nextTick(() => {
  console.log('Next tick callback!');
});
```

<div style={{ padding: '5px' }}>
  <h5>queueMicrotask()</h5>
  <p>
    This method allows you to schedule a microtask that will execute after the
    current operation completes but before any macrotasks are processed
  </p>
</div>

```js
queueMicrotask(() => {
  console.log('Microtask executed!');
});
```

<div style={{ padding: '5px' }}>
  <h5>MutationObserver callbacks</h5>
  <p>
    These are invoked after detecting changes in the DOM structure, such as
    element additions or removals. They are processed as microtasks after the
    synchronous code execution
  </p>
</div>

```js
const observer = new MutationObserver(() => {
  console.log('DOM changed!');
});
```

<div style={{ marginTop: '20px' }}>
  <h3>The Execution Order</h3>
  <div style={{ padding: '5px' }}>
    <p>
      <strong>Understanding the execution order</strong> is essential here's how
      Node.js processes tasks:
    </p>
    <ul>
      <li>
        🥇<strong>Execute</strong> all synchronous code
      </li>
      <li>
        🥈<strong>Execute</strong> all tasks in the microtask queue
      </li>
      <li>
        🥉 <strong>Execute</strong> one task from the macrotask queue
      </li>
      <li>
        ⤴️<strong>Return</strong> to step 2
      </li>
    </ul>
  </div>
</div>

```js
console.log('Script start');

setTimeout(() => console.log('Timeout 1'), 0);

Promise.resolve()
  .then(() => console.log('Promise 1'))
  .then(() => console.log('Promise 2'));

setTimeout(() => console.log('Timeout 2'), 0);

console.log('Script end');

// Output:
// Script start
// Script end
// Promise 1
// Promise 2
// Timeout 1
// Timeout 2
```

<div style={{ padding: '5px' }}>
  <h5>The process.nextTick() Special Case</h5>
  <p>
    In Node.js, <strong>process.nextTick()</strong> is a special case that takes
    priority over all other microtasks. It will execute immediately after the
    current synchronous operation completes, even before other microtasks:
  </p>
</div>

```js
Promise.resolve().then(() => console.log('Promise'));
process.nextTick(() => console.log('nextTick'));
setTimeout(() => console.log('setTimeout'), 0);

// Output:
// nextTick
// Promise
// setTimeout
```

<div style={{ marginTop: '20px' }}>
  <h3>Conclusion</h3>
  <div style={{ padding: '5px' }}>
    <p>
      The event loop is key to handling asynchronous operations in Node.js. By
      understanding how microtasks and macrotasks work, developers can better
      manage task execution and improve application performance. This leads to
      faster, more responsive applications that handle multiple tasks without
      blocking the main thread.
    </p>
  </div>
</div>


NodeJS MacroTasks and MicroTasks


<div style={{ marginTop: "20px" }}>
  <h3>Introduction</h3>
  <div style={{ padding: "5px" }}>
 <p>
  When dealing with <strong>large file uploads</strong>, <strong>efficiency</strong> and <strong>speed</strong> are crucial. 
  AWS S3 provides two powerful features to help with this: <strong>Multipart Upload</strong> and <strong>Transfer Acceleration</strong>. 
  In this blog post, we'll explore how to use these features with <strong>Node.js</strong> to optimize your <strong>file upload process</strong>.
</p>
 </div>
</div>

<div style={{ marginTop: "20px" }}>
  <h3>What is Multipart Upload?</h3>
  <div style={{ padding: "5px" }}>
  <p>
  <strong>Multipart upload</strong> lets you break a large file into smaller, manageable parts and upload them individually.
   Each part uploads independently, often in <strong>parallel</strong>, which can drastically speed up the process. 
   If any part fails, you only need to retry that specific part, saving you from starting the entire upload over again.
</p>

  <img src="/images/posts/s3-data-send/1.png" style={{ marginTop: "5px" }} />

  <p>
  You should use <strong>multipart upload</strong> when working with <strong>large files</strong> that exceed the <strong>5 GB</strong> limit for a single upload in AWS S3. Multipart upload is ideal for files that are too large to upload in a single request or when uploads might take a long time. By splitting the file into smaller parts (each part can range from <strong>5 MB to 5 GB</strong>), you can <strong>improve upload speed</strong> and handle failures more efficiently. In case of a failure, only the specific part needs to be retried, and you can even upload parts in <strong>parallel</strong> for better performance.
</p>

 </div>
</div>

<div style={{ marginTop: "20px" }}>
  <h3>What is Transfer Acceleration?</h3>
  <div style={{ padding: "5px" }}>
   <p>
  <strong>Transfer Acceleration</strong> uses Amazon CloudFront's globally distributed edge locations to speed up uploads to S3. 
  This feature helps to <strong>reduce latency</strong> and significantly <strong>improve upload speeds</strong>, 
  making it especially beneficial for users located far from the S3 bucket's region.
</p>

  </div>
    <img src="/images/posts/s3-data-send/2.png" style={{ marginTop: "5px" }} />
    <p><strong>Transfer Acceleration</strong> works by routing traffic through the nearest <strong>edge location</strong>, reducing <strong>latency</strong> and improving <strong>upload speeds</strong>. You can use <strong>Transfer Acceleration</strong> when you need to upload large files quickly from remote locations, or when you have users distributed <strong>globally</strong>. However, it's important to note that <strong>Transfer Acceleration</strong> comes at an additional <strong>cost</strong>. The pricing depends on the <strong>data transfer amount</strong>, with higher costs for longer distances between the client and the bucket. It’s best to use this feature when faster upload speeds justify the extra cost, particularly for <strong>time-sensitive</strong> or <strong>large-scale uploads</strong>.</p>

</div>

<div style={{ marginTop: "20px" }}>
  <h3>From Concept to Execution</h3>
  <div style={{ padding: "5px" }}>
<p>
  In this post, I’ll show you how to create a <strong>Node.js service</strong> that makes it easy to upload <strong>large files</strong>. We’ll use <strong>multipart upload</strong>, add an option for <strong>Transfer Acceleration</strong>, and include a <strong>retry feature</strong> to handle any parts that fail.
</p>

  <img src="/images/posts/s3-data-send/3.png" style={{ marginTop: "5px" }} />
 
 </div>
</div>

<div style={{ marginTop: "20px" }}>
  <h3>Setting Up</h3>
  <div style={{ padding: "5px" }}>
   <p>First, create a folder named S3-Upload-Service and install the Node.js AWS SDK, if it isn't already installed</p>
 </div>
</div>

```cmd
mkdir S3-Upload-Service
cd S3-Upload-Service
npm init
npm install @aws-sdk/client-s3
```

<div style={{ marginTop: "20px" }}>
  <div style={{ padding: "5px" }}>
   <p>
  The AWS SDK is used to interact with AWS services and make API calls from your application.
  However, it requires proper configuration, including providing credentials, to authenticate and grant access to AWS resources.
   </p>
     <p>
       To configure the credentials, you simply need to create an IAM user and obtain the access and secret keys. These keys will be used by the SDK when making API calls. 
       Keep in mind that the SDK's access is limited by the permissions assigned to the IAM user.
     </p>
      <p>
  Now that our environment is set up, we can begin implementing the code logic. Let's start by creating a file named <strong>index.js</strong> and adding the following code:
</p>
 </div>
</div>

```js
// index.js
const {
  S3Client,
  CreateMultipartUploadCommand,
  UploadPartCommand,
  CompleteMultipartUploadCommand,
} = require("@aws-sdk/client-s3");

const fs = require("fs");
const path = require("path");

class S3MultipartUploader {
  constructor(region = "us-east-1", accessKeyId, secretAccessKey) {
    this.s3Client = new S3Client({
      credentials: {
        accessKeyId,
        secretAccessKey,
      },
      region,
    });
  }
  
  // this function will contain the upload logic
  async uploadFile(filePath, bucketName) {}
}
```
<div style={{ marginTop: "20px" }}>
  <div style={{ padding: "5px" }}>
<p>
  The code above defines a class designed to manage file uploads to S3. By creating an instance of this class, 
  you can call the <strong>uploadFile</strong> function, which will handle the upload logic. 
  The configuration for the AWS SDK is managed in the constructor, where essential details such as 
  <strong>region</strong>, <strong>accessKeyId</strong>, and <strong>secretAccessKey</strong> are provided.
</p>
<p>
  Now it’s time for the best part: implementing the <strong>uploadFile</strong> function
</p>
  </div>
</div>

```js
 async uploadFile(filePath, bucketName) {
    try{
      // File details
      const fileSize = fs.statSync(filePath).size;
      const fileName = path.basename(filePath);

      // Multipart upload configuration
      let partSize = 5 * 1024 * 1024; // 5 MB per part
      const numberOfParts = Math.ceil(fileSize / partSize);

      // Initiate multipart upload
      
     // Upload parts

    // Wait for all parts to upload

    // Sort parts to ensure correct order

    // Complete multipart upload

    }catch (error) {
      console.error("Error during multipart upload:", error);
      throw error;
    }
  }
```

<div style={{ marginTop: "20px" }}>
  <div style={{ padding: "5px" }}>
  <p>
  This code determines the <strong>file size</strong> and <strong>name</strong>, configures each upload part to be <strong>5 MB</strong> (the min size file need to be 5M), and calculates the <strong>number of parts</strong> needed.
</p>
<p>Next, we will go through the remaining steps outlined in the code snippet using comments, explaining each one in detail starting by the <strong>Initiate multipart upload</strong>.</p>
  </div>
</div>

```js
// Initiate multipart upload
const multipartUpload = await this.s3Client.send(
  new CreateMultipartUploadCommand({
    Bucket: bucketName,
    Key: fileName,
  })
);

console.log(`Multipart upload initiated for ${fileName}`);

```

<div style={{ marginTop: "20px" }}>
<div style={{ padding: "5px" }}>
  <p><strong>CreateMultipartUploadCommand</strong> is used to initiate a multipart upload to an S3 bucket.</p>
  <p>It starts the upload process and returns an <strong>UploadId</strong> needed for uploading parts.</p>
  <p>This command is useful for efficiently uploading <strong>large files</strong> in smaller parts.</p>
  <p>It helps in resuming uploads and managing file uploads in <strong>parallel</strong> that's why we will use it in the next steps.</p>
</div>
</div>

```js
// Upload parts
const uploadPromises = [];
const uploadedParts = [];

for (let partNumber = 1; partNumber <= numberOfParts; partNumber++) {
  const start = (partNumber - 1) * partSize;
  const end = Math.min(start + partSize, fileSize);
  partSize = end - start;

  const uploadPromise = new Promise((resolve, reject) => {
    const readStream = fs.createReadStream(filePath, {
      start,
      end: end - 1,
    });

    this.s3Client
      .send(
        new UploadPartCommand({
          Bucket: bucketName,
          Key: fileName,
          PartNumber: partNumber,
          UploadId: multipartUpload.UploadId,
          Body: readStream,
        })
      )
      .then((uploadPartResult) => {
        uploadedParts.push({
          ETag: uploadPartResult.ETag,
          PartNumber: partNumber,
        });
        resolve();
      })
      .catch(reject);
  });

  uploadPromises.push(uploadPromise);
}

```
<div style={{ marginTop: "20px" }}>
<div style={{ padding: "5px" }}>
<p>The code above divides the file into parts, each of size 5MB, using the <strong>fs.createReadStream</strong> function, a built-in Node.js method and creates a <strong>Promise</strong> for uploading a part of a file to S3 using the <strong>UploadPartCommand</strong>.</p>
<p>A <strong>readStream</strong> is created using the <strong>fs.createReadStream</strong> function, which reads a specific range of the file defined by <strong>start</strong> and <strong>end</strong> values.</p>
<p>The <strong>UploadPartCommand</strong> is then sent to S3 with the <strong>Bucket</strong> name, <strong>file name (Key)</strong>, <strong>PartNumber</strong>, <strong>UploadId</strong>, and the <strong>Body</strong> (which is the read stream). Once the part is uploaded, its <strong>ETag</strong> and <strong>PartNumber</strong> are pushed into the <strong>uploadedParts</strong> array to track the upload status.</p>
<p>The <strong>uploadPromise</strong> is added to an array of promises <strong>uploadPromises</strong> to manage multiple part uploads concurrently.</p>
<p><strong>Note:</strong> We are creating multiple promises that will be executed in parallel in the next step.</p>
</div>
</div>

```js
// Wait for all parts to upload
await Promise.all(uploadPromises);

// Sort parts to ensure correct order
uploadedParts.sort((a, b) => a.PartNumber - b.PartNumber);
```
<div style={{ marginTop: "20px" }}>
<div style={{ padding: "5px" }}>
<p>After all parts are uploaded, we need to sort them because the upload process is parallel, and any part may finish before another. The sorted parts will then be used to complete the multipart upload process in the next step.</p>
</div>
</div>

```js
// Complete multipart upload
const completeMultipartUploadResult = await this.s3Client.send(
  new CompleteMultipartUploadCommand({
    Bucket: bucketName,
    Key: fileName,
    UploadId: multipartUpload.UploadId,
    MultipartUpload: { Parts: uploadedParts },
  })
);

console.log(
  "File uploaded successfully:",
  completeMultipartUploadResult.Location
);
return completeMultipartUploadResult;
```

<div style={{ marginTop: "20px" }}>
<div style={{ padding: "5px" }}>
<p>In this step, we inform S3 that all parts have been uploaded, so it can complete the multipart upload process by providing the <strong>ETag</strong> and <strong>PartNumber</strong> for each uploaded part.</p>
<p>At this point, we are good, but we need to add a retry mechanism that will be executed if any part fails. We begin by adding a new function called <strong>retry</strong> inside the class.</p>
</div>
</div>

```js
 async retry(fn, args, retryCount = 0) {
    try {
      return await fn(...args);
    } catch (error) {
      if (retryCount < 3) {
        console.log(`Attempt ${retryCount + 1} failed. Retrying...`);
        await new Promise(resolve => setTimeout(resolve, 1000 * Math.pow(2, retryCount))); // Exponential backoff
        return this.retry(fn, args, retryCount + 1);
      }
      throw error;
    }
  }
 ```

<div style={{ marginTop: "20px" }}>
<div style={{ padding: "5px" }}>
<p>The retry function will attempt to upload a file up to three times if the upload fails, using the Exponential Backoff technique. 
After implementing this function, we will need to update the upload parts step accordingly.</p>
</div>
</div>

```js
 // Upload parts
const uploadPromises = [];
const uploadedParts = [];

for (let partNumber = 1; partNumber <= numberOfParts; partNumber++) {
  const start = (partNumber - 1) * partSize;
  const end = Math.min(start + partSize, fileSize);
  partSize = end - start;

  const uploadPromise = new Promise((resolve, reject) => {
    const readStream = fs.createReadStream(filePath, {
      start,
      end: end - 1,
    });

    // Upload part with retry mechanism
    this.retry(this.s3Client.send.bind(this.s3Client), [
      new UploadPartCommand({
        Bucket: bucketName,
        Key: fileName,
        PartNumber: partNumber,
        UploadId: multipartUpload.UploadId,
        Body: readStream,
      }),
    ])
      .then((uploadPartResult) => {
        uploadedParts.push({
          ETag: uploadPartResult.ETag,
          PartNumber: partNumber,
        });
        resolve();
      })
      .catch(reject);
  });

  uploadPromises.push(uploadPromise);
}
 ```

<div style={{ marginTop: "20px" }}>
<div style={{ padding: "5px" }}>
<p>With this, we can say that our <strong>uploadFile</strong> function supports the <strong>retry mechanism</strong> if any part fails.
 The last thing we need to add is the ability to use <strong>S3 Transfer Acceleration</strong>.</p>
<p>To use <strong>S3 Transfer Acceleration</strong>, we need to update the <strong>constructor</strong> since this functionality is optional. A variable must be passed to indicate whether the class should enable it.
 Additionally, you need to activate the transfer acceleration feature for your bucket in the AWS Management Console. Without enabling this feature, it will not work.</p>
</div>
</div>

```js
    constructor(region = 'us-east-1', options = {}) {
        const { 
          accessKeyId, 
          secretAccessKey, 
          useTransferAcceleration 
        } = options;
    
        const clientConfig = {
          credentials: {
            accessKeyId: accessKeyId,
            secretAccessKey: secretAccessKey
          },
          region: region,
          useAccelerateEndpoint: useTransferAcceleration,
        };
    
        this.s3Client = new S3Client(clientConfig);
    }
 ```

 <div style={{ marginTop: "20px" }}>
<div style={{ padding: "5px" }}>
<p>After all this effort, we can confidently say that all the steps are complete, and our multipart upload is ready to use. Now, let's test it and observe the results!</p>
</div>
</div>


```js
async function main() {
  const options = {
    accessKeyId: "Your accessKeyId",
    secretAccessKey: "Your secretAccessKey",
    useTransferAcceleration: true,
  };

  // Create uploader with transfer acceleration
  const uploader = new S3MultipartUploader("eu-central-1", options);
  await uploader.uploadFile(
    "TestFile.pdf", // Local file path with a size of 5.8 MB
    "doodooti" // S3 Bucket Name
  );
}

main();
```

<center><img src="/images/posts/s3-data-send/4.png" style={{ marginTop: "5px" }} /></center>
<center><small>It works 💥 </small></center>
<center><img src="/images/posts/s3-data-send/5.png" style={{ marginTop: "5px" }} /></center>

<div style={{ marginTop: "20px" }}>
  <h3>Conclusion</h3>
  <div style={{ padding: "5px" }}>
<p>
  Using AWS S3's <strong>multipart upload</strong> and <strong>transfer acceleration</strong> features can greatly
  enhance the <strong>efficiency</strong> and <strong>speed</strong> of your file uploads. 
  This example demonstrates how to implement these features in a <strong>Node.js application</strong>, 
  ensuring your uploads are <strong>fast</strong> and <strong>reliable</strong>.
</p>
 <p>Code : https://github.com/ridhamz/nodejs-s3-files-uploads </p>
  </div>
</div>


Optimizing S3 File Uploads with NodeJS


<div style={{ marginTop: '20px' }}>
  <h3>Introduction</h3>
  <div style={{ padding: '5px' }}>
    <p>
    In cloud infrastructure, <strong>security</strong> is essential, and a <strong>bastion host</strong> is key to keeping your private network safe. Acting as a secure gateway, it protects access to <strong>private resources</strong>. In this post, we’ll explore AWS EC2 bastion hosts, their role in <strong>network security</strong>, and the best ways to set them up for maximum security.
</p>

  </div>
</div>

<div style={{ marginTop: '20px' }}>
  <h3>What a Bastion Host Is and How It Works</h3>
  <div style={{ padding: '5px' }}>
    <p>
      Simply put, a <strong>bastion host</strong> (also called a “jump box”) is
      a server specifically designed to provide{' '}
      <strong>secure, limited access</strong> to your private network. Instead
      of allowing direct access to your servers, which increases vulnerability,
      a bastion host acts as a go-between.
    </p>
  </div>
  <center><img src="/images/posts/aws-bastian-host/1.png" /></center>
   <div style={{ padding: '5px' }}>
    <p>
    Think of a <strong>bastion host</strong> as a secure gateway or checkpoint into your <strong>private network</strong>. It's an <strong>EC2 instance</strong> that sits in a <strong>public subnet</strong> and acts as the only entry point to your private EC2 instances. When you need to access a private server, you first <strong>SSH</strong> into the bastion host, and from there, you can SSH into your private servers. 
</p>
<p>
    Imagine a <strong>bastion host</strong> as a secure entry point to your <strong>private network</strong>. It’s an <strong>EC2 instance</strong> in a public subnet that acts as the only door to reach your private servers. If you need to access a server in the private network, you start by connecting to the bastion host, and then from there, you can connect to the private servers.
</p>

<p>
  The bastion host is set up with <strong>strict security rules</strong>,
  usually allowing only <strong>SSH</strong> access through{' '}
  <strong>port 22</strong> and accepting connections only from approved{' '}
  <strong>IP addresses</strong>, this way your <strong>private servers</strong>{' '}
  stay protected from the public internet, but authorized admins can still
  access them securely.
</p>

  </div>
</div>

<div style={{ marginTop: '20px' }}>
  <h3>From Concept to Execution</h3>
  <div style={{ padding: '5px' }}>
    <p>
      Throughout this post, I will guide you through the architecture outlined
      below in order to build a Bastion Host.
    </p>
  </div>
</div>
<center>
  <img src="/images/posts/aws-bastian-host/2.png" />
</center>
<div style={{ padding: '5px' }}>
  <p>
    The idea is to provision <strong>two EC2 servers</strong> in a{' '}
    <strong>private subnet</strong> and another <strong>EC2 server</strong> in a{' '}
    <strong>public subnet</strong> to serve as a <strong>jump box</strong>. The{' '}
    <strong>jump box</strong> will allow the <strong>admin</strong> to securely
    access the <strong>private instances</strong> for{' '}
    <strong>configuration</strong> and other administrative tasks.
  </p>
</div>

<div style={{ marginTop: '20px' }}>
  <h3>Provision Resources</h3>
  <div style={{ padding: '5px' }}>
    <p>
      I’ll use a CloudFormation template to set up the VPC, subnets, the
      internet gateway and route tables. For the EC2 instances, I’ll set them up
      through the AWS Console to make things easier to follow.
    </p>
    <p>
      First, create a file named <strong>BastionHosts.yaml</strong> and start by
      adding the following code:
    </p>
  </div>
</div>

```yaml
AWSTemplateFormatVersion: '2010-09-09'
Description: 'VPC with Public and Private Subnets'

Parameters:
  VpcCidr:
    Type: String
    Default: '10.0.0.0/16'
    Description: CIDR block for the VPC

  PublicSubnetCidr:
    Type: String
    Default: '10.0.1.0/24'
    Description: CIDR block for Public Subnet

  PrivateSubnetCidr:
    Type: String
    Default: '10.0.2.0/24'
    Description: CIDR block for Private Subnet
```

<div style={{ padding: '5px' }}>
<p>This is the <strong>Parameters section</strong> of a <strong>CloudFormation template</strong>. 
It defines three parameters to configure the IP ranges for a VPC and its subnets:</p>

<ol>
  <li style={{ marginTop: '5px' }}>
    <strong>VpcCidr</strong>: Sets the IP range (CIDR block) for the entire VPC.
    Default Value: <code>10.0.0.0/16</code>
  </li>
  <li style={{ marginTop: '5px' }}>
    <strong>PublicSubnetCidr</strong>: Sets the IP range for the public subnet.
    Default Value: <code>10.0.1.0/24</code>
  </li>
  <li style={{ marginTop: '5px' }}>
    <strong>PrivateSubnetCidr</strong>: Sets the IP range for the private
    subnet. Default Value: <code>10.0.2.0/24</code>
  </li>
</ol>

<p>These parameters make the VPC’s IP ranges easy 
to customize without changing the template code directly.</p>
<p>Next, we need to add the Resources section, which will define the logical 
resources that will become physical resources when the template is deployed.</p>
</div>

```yaml
Resources:
  # VPC
  MzVPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: !Ref VpcCidr
      EnableDnsHostnames: true
      EnableDnsSupport: true
      Tags:
        - Key: Name
          Value: Mz VPC

  # Internet Gateway
  InternetGateway:
    Type: AWS::EC2::InternetGateway
    Properties:
      Tags:
        - Key: Name
          Value: Mz IGW

  AttachGateway:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      VpcId: !Ref MzVPC
      InternetGatewayId: !Ref InternetGateway
```

<div style={{ padding: '5px' }}>
<p>This section defines the <strong>resources</strong> that will be created when the CloudFormation template is deployed:</p>

<ol>
  <li style={{ marginTop: '5px' }}>
    <strong>MzVPC</strong>: Creates a <strong>VPC</strong> with the CIDR block
    provided by the <code>VpcCidr</code> parameter. It also enables DNS
    hostnames and support for DNS.
  </li>
  <li style={{ marginTop: '5px' }}>
    <strong>InternetGateway</strong>: Creates an{' '}
    <strong>Internet Gateway</strong> (IGW) to provide internet access to the
    VPC.
  </li>
  <li style={{ marginTop: '5px' }}>
    <strong>AttachGateway</strong>: Attaches the{' '}
    <strong>Internet Gateway</strong> to the <strong>VPC</strong> so that
    resources in the VPC can access the internet.
  </li>
</ol>

<p>
  These resources define the basic network infrastructure: a VPC, internet
  access, and the necessary connection between them.
</p>
</div>

```yaml
# Public Subnet
PublicSubnet:
  Type: AWS::EC2::Subnet
  Properties:
    VpcId: !Ref MzVPC
    CidrBlock: !Ref PublicSubnetCidr
    AvailabilityZone: !Select [0, !GetAZs '']
    MapPublicIpOnLaunch: true
    Tags:
      - Key: Name
        Value: Public Subnet

# Private Subnet
PrivateSubnet:
  Type: AWS::EC2::Subnet
  Properties:
    VpcId: !Ref MzVPC
    CidrBlock: !Ref PrivateSubnetCidr
    AvailabilityZone: !Select [0, !GetAZs '']
    Tags:
      - Key: Name
        Value: Private Subnet
```

<div style={{ padding: '5px' }}>
<p>The code above defines the <strong>public subnet</strong> and <strong>private subnet</strong> that will be created in the VPC:</p>

<ol>
  <li style={{ marginTop: '5px' }}>
    <strong>PublicSubnet</strong>: Creates a <strong>public subnet</strong>{' '}
    within the VPC using the CIDR block provided by the{' '}
    <code>PublicSubnetCidr</code> parameter. It is placed in the first available
    availability zone and enables public IP addresses for instances launched in
    this subnet.
  </li>
  <li style={{ marginTop: '5px' }}>
    <strong>PrivateSubnet</strong>: Creates a <strong>private subnet</strong>{' '}
    within the VPC using the CIDR block provided by the{' '}
    <code>PrivateSubnetCidr</code> parameter. It is also placed in the first
    available availability zone, but it does not automatically assign public IPs
    to instances launched in this subnet.
  </li>
</ol>

<p>
  These subnets define the network layout: one for public-facing resources and
  one for private, internal resources.
</p>

</div>

```yaml
# Public Route Table
PublicRouteTable:
  Type: AWS::EC2::RouteTable
  Properties:
    VpcId: !Ref MzVPC
    Tags:
      - Key: Name
        Value: Public Route Table

PublicRoute:
  Type: AWS::EC2::Route
  DependsOn: AttachGateway
  Properties:
    RouteTableId: !Ref PublicRouteTable
    DestinationCidrBlock: 0.0.0.0/0
    GatewayId: !Ref InternetGateway

PublicSubnetRouteTableAssociation:
  Type: AWS::EC2::SubnetRouteTableAssociation
  Properties:
    SubnetId: !Ref PublicSubnet
    RouteTableId: !Ref PublicRouteTable

# Private Route Table
PrivateRouteTable:
  Type: AWS::EC2::RouteTable
  Properties:
    VpcId: !Ref MzVPC
    Tags:
      - Key: Name
        Value: Private Route Table

PrivateSubnetRouteTableAssociation:
  Type: AWS::EC2::SubnetRouteTableAssociation
  Properties:
    SubnetId: !Ref PrivateSubnet
    RouteTableId: !Ref PrivateRouteTable
```

<div style={{ padding: '5px' }}>
<p>The code above defines the route tables for both the <strong>public</strong> and <strong>private</strong> subnets, as well as the routes that connect them to the internet:</p>

<ol>
  <li style={{ marginTop: '5px' }}>
    <strong>PublicRouteTable</strong>: Creates a <strong>route table</strong>{' '}
    for the public subnet and associates it with the VPC.
  </li>
  <li style={{ marginTop: '5px' }}>
    <strong>PublicRoute</strong>: Defines the route for the public subnet,
    allowing traffic to the internet via the <strong>Internet Gateway</strong>.
    The route is set to <code>0.0.0.0/0</code>, meaning it allows all traffic to
    the internet.
  </li>
  <li style={{ marginTop: '5px' }}>
    <strong>PublicSubnetRouteTableAssociation</strong>: Associates the public
    subnet with the <strong>public route table</strong> so that instances in the
    public subnet can route traffic to the internet.
  </li>
  <li style={{ marginTop: '5px' }}>
    <strong>PrivateRouteTable</strong>: Creates a <strong>route table</strong>{' '}
    for the private subnet, again associating it with the VPC but without a
    direct route to the internet.
  </li>
  <li style={{ marginTop: '5px' }}>
    <strong>PrivateSubnetRouteTableAssociation</strong>: Associates the private
    subnet with the <strong>private route table</strong>, ensuring instances in
    the private subnet route traffic only internally.
  </li>
</ol>

<p>
  These route tables and associations ensure that the public subnet can access
  the internet while the private subnet remains isolated or has controlled
  access to the internet.
</p>
</div>

<div style={{ padding: '5px' }}>
  <p>
    Now that our template is ready, let's deploy it using the CloudFormation
    service in the AWS Console UI. If you click the View in Infrastructure
    Composer button, you will see the infrastructure as code that we created, as
    represented in the image below
  </p>
</div>
<center>
  <img src="/images/posts/aws-bastian-host/3.png" />
</center>

<div style={{ padding: '5px' }}>
  <p>
    Now we're ready to proceed with deploying our template. Simply wait until
    the CloudFormation stack status updates to <strong>CREATE_COMPLETE</strong>
    .{' '}
  </p>
</div>

<center>
  <img src="/images/posts/aws-bastian-host/18.png" />
  <small>Stack created successfully</small>
</center>

<center>
  <img src="/images/posts/aws-bastian-host/4.png" />
  <small>Our physical resources are now ready</small>
</center>

<div style={{ padding: '5px' }}>
  <p>
    {' '}
    Now that our VPC architecture is ready, we can begin creating the EC2 instances
    to complete the Bastion Host setup.{' '}
  </p>
</div>

<div style={{ marginTop: '20px' }}>
  <h3>Provision The Bastian Host</h3>
  <div style={{ padding: '5px' }}>
    <p>
      As mentioned in the previous section, I will use the Console UI to
      provision the EC2 instances. Go to the EC2 service, and click on Launch
      Instance to begin.
    </p>
    <p>
      After entering the name and selecting the type for the EC2 instance, we
      need to specify a key pair to enable SSH access from our computer to the
      Bastion Host.
    </p>
    <center>
      <img
        src="/images/posts/aws-bastian-host/7.png"
        style={{ marginTop: '5px' }}
      />
    </center>
    <p style={{ marginTop: '5px' }}>
      If you don't already have a key pair, simply click on the Create new key
      pair button
    </p>
    <center>
      <center>
        <img
          src="/images/posts/aws-bastian-host/5.png"
          style={{ marginTop: '5px' }}
        />
      </center>
    </center>
    <p style={{ marginTop: '5px' }}>
      Under the network settings, we need to select the VPC we created and the
      public subnet, as the jump box needs to be located in the public subnet
    </p>
    <center>
      <img
        src="/images/posts/aws-bastian-host/8.png"
        style={{ marginTop: '5px' }}
      />
    </center>
    <p style={{ marginTop: '5px' }}>
      The final step is to set the security group, we will create a new one and
      allow SSH connections from anywhere
    </p>
    <center>
      <img
        src="/images/posts/aws-bastian-host/9.png"
        style={{ marginTop: '5px' }}
      />
    </center>
    <p style={{ marginTop: '5px' }}>
      <b>Note:</b> We can specify a specific CIDR IP block in the source of the
      security group for better security, but for the purposes of this post, we
      will allow access from anywhere.
    </p>
    <p style={{ marginTop: '5px' }}>
      After that, click on the Launch Instance button and wait for the EC2
      instance to be provisioned. Once the instance is running, go to the
      Instance Details section to verify the Public IP other configurations.
    </p>
    <center>
      <img
        src="/images/posts/aws-bastian-host/11.png"
        style={{ marginTop: '5px' }}
      />
    </center>
  </div>
</div>

<div style={{ marginTop: '20px' }}>
  <h4>SSH the Bastian Host</h4>
  <div style={{ padding: '5px' }}>
    <p>
      In my case, the public IP address for the Bastion Host is "35.91.217.143,"
      and the key name is "MZ-Bastina-Host-Key.pem", so to establish an SSH
      connection, follow these steps:
    </p>
    <ol>
      <li style={{ marginTop: '5px' }}>
        <strong>Open the terminal</strong>: Navigate to the folder where you
        downloaded the key pair.
      </li>
      <li style={{ marginTop: '5px' }}>
        <strong>Change the permissions for the key pair</strong>: Type the
        following command in the terminal:
        <br />
        <code>chmod 400 "MZ-Bastina-Host-Key.pem"</code>
      </li>
      <li style={{ marginTop: '5px' }}>
        <strong>Copy the public IP address</strong>: Copy the public IP address
        of the Bastion Host.
      </li>
      <li style={{ marginTop: '5px' }}>
        <strong>Run the SSH command</strong>: Run the following SSH command to
        connect:
        <br />
        <code>ssh -i .\MZ-Bastina-Host-Key.pem ec2-user@35.91.217.143</code>
      </li>
    </ol>
  </div>
</div>
<center>
  <img src="/images/posts/aws-bastian-host/10.png" />
</center>
<center>
  <small>Successfully connected to the Jump Box 🥳</small>
</center>

<div style={{ marginTop: '20px' }}>
  <h3>Provision The Private Instances</h3>
  <div style={{ padding: '5px' }}>
    <p>
      The steps to provision the private instances, <strong>Server 1</strong>{' '}
      and <strong>Server 2</strong>, are the same as those for the{' '}
      <strong>Bastion Host</strong> instance. The only difference is in the{' '}
      <strong>Network settings</strong>, as these instances are{' '}
      <strong>private</strong> and need to be located in the{' '}
      <strong>private subnet</strong> without enabling the{' '}
      <strong>Auto-assign public IP</strong> option.
    </p>
  </div>
</div>
<center>
  <img src="/images/posts/aws-bastian-host/12.png" />
</center>

<div style={{ padding: '5px' }}>
  <p>
    About the <strong>security group</strong>, it will be different because we
    need to allow <strong>SSH access</strong> only from the{' '}
    <strong>Jump Box</strong>. We can do this by changing the{' '}
    <strong>source type</strong> to <strong>custom</strong>, then selecting the{' '}
    <strong>security group</strong> of the Jump Box as the source.
  </p>
</div>
<center>
  <img src="/images/posts/aws-bastian-host/13.png" />
</center>
<div style={{ padding: '5px' }}>
  <p>
    Now that our three <strong>instances</strong> are running, it's time to test
    the <strong>connectivity</strong> from the <strong>Jump Box</strong> to the
    private instances, <strong>Server 1</strong> and <strong>Server 2</strong>.
  </p>
</div>
<center>
  <img src="/images/posts/aws-bastian-host/14.png" />
</center>
<div style={{ padding: '5px' }}>
  <p>
    In order to do an SSH connection from our Bastion Host to the private
    servers, we first need to upload the key pairs of Server 1 and Server 2 to
    the Bastion Host and we can use the <strong>scp</strong> command.
  </p>
</div>
<center>
  <img src="/images/posts/aws-bastian-host/15.png" />
</center>

<div style={{ padding: '5px' }}>
  <p>
    Now, let's try to make the SSH connection from the jump box. For that, we
    need to use the private IPs of Server 1 (ip: 10.0.2.252) and Server 2 (ip:
    10.0.2.164).
  </p>
</div>
<center>
  <img src="/images/posts/aws-bastian-host/16.png" />
  <small>Successfully connected to the Server 1 🥳</small>
</center>

<center>
  <img src="/images/posts/aws-bastian-host/17.png" />
  <small>Successfully connected to the Server 2 🥳</small>
</center>

<div style={{ marginTop: '20px' }}>
  <h3>Conclusion</h3>
  <div style={{ padding: '5px' }}>
    <p>
      In summary, setting up a Bastion Host is a key step in building secure
      access points within your AWS environment. By carefully configuring
      networking, security groups, and SSH keys, we’ve established a secure
      gateway that allows controlled access to private instances. This setup
      enhances security by isolating access to sensitive resources, while also
      simplifying management and reducing potential vulnerabilities.
    </p>
  </div>
</div>


AWS EC2 Bastion Hosts


<div style={{ marginTop: "20px" }}>
  <h3>Introduction</h3>
  <div style={{ padding: "5px" }}>
    <p>
      <strong>TypeScript decorators</strong> are a powerful feature that allows you to add 
      <strong>metadata</strong> to classes, methods, properties, and parameters. They provide 
      a way to modify or enhance the behavior of your code without changing its structure.
    </p>
  </div>
  <center> <img src="/images/posts/ts-deco/1.png" alt="decorators" /></center>
    <div style={{ padding: "5px" }}>
     <p>
      <strong>Decorators</strong> are a <strong>stage 3 proposal</strong> for JavaScript and are available 
      as an <strong>experimental feature</strong> in TypeScript. They are 
      <strong>functions</strong> that can be attached to classes, methods, properties, 
      or parameters to modify their behavior or add <strong>metadata</strong>.
    </p>
    </div>
   <div style={{ padding: "5px" }}>
      <p>
      To use <strong>decorators</strong> in TypeScript, you need to enable the 
      <strong>experimentalDecorators</strong> compiler option in your <strong>tsconfig.json</strong>:
    </p>
   </div>

</div>

```typescript
{
  "compilerOptions": {
    "experimentalDecorators": true
  }
}
```

<div style={{ marginTop: '20px' }}>
  <h3>Class Decorators</h3>
  <div style={{ padding: '5px' }}>
    <p>
      Class decorators are applied to the <strong>constructor</strong> of the
      class and can be used to <strong>observe</strong>, <strong>modify</strong>,
      or <strong>replace</strong> a <strong>class definition</strong>.
    </p>
    <h5>Example: Singleton Pattern</h5>
  </div>
</div>

```typescript
function Singleton<T extends { new (...args: any[]): {} }>(constructor: T) {
  let instance: T;

  return class extends constructor {
    constructor(...args: any[]) {
      if (instance) {
        return instance;
      }
      super(...args);
      instance = this;
    }
  };
}
```

<div style={{ padding: '5px' }}>
  <p>
    This is a <strong>class decorator</strong> that implements the{' '}
    <strong>Singleton pattern</strong>. It returns a <strong>new class</strong>{' '}
    that <strong>extends</strong> the <strong>original constructor</strong>. The
    new class ensures only <strong>one instance</strong> is created by{' '}
    <strong>checking</strong> if an instance already <strong>exists</strong>{' '}
    before creating a <strong>new one</strong>.
  </p>
</div>

```typescript
@Singleton
class Database {
  constructor(private readonly url: string) {
    console.log(`Connecting to ${url}...`);
  }
}
```

<div style={{ padding: '5px' }}>
  <p>
    This is the class we're applying the <strong>Singleton pattern</strong> to.
    It has a <strong>private constructor</strong> (to prevent{' '}
    <strong>direct instantiation</strong>) and a <strong>connect method</strong>
  </p>
</div>

```typescript
const db1 = new Database('MySQL');
const db2 = new Database('PostgreSQL');

console.log(db1 === db2); // true
db1.connect(); // "Connected to MySQL"
db2.connect(); // "Connected to MySQL"
```

<div style={{ padding: '5px' }}>
  <p>
    This demonstrates the <strong>Singleton behavior</strong>. Despite trying to
    create <strong>two instances</strong> with <strong>different names</strong>,
    we end up with the <strong>same instance</strong>. Both <strong>db1</strong>{' '}
    and <strong>db2</strong> refer to the <strong>same object</strong> (the{' '}
    <strong>first one created</strong>), which is why they both{' '}
    <strong>connect</strong> to "<strong>MySQL</strong>".
  </p>
</div>

<div style={{ marginTop: '20px' }}>
  <h3>Method Decorators</h3>
  <div style={{ padding: '5px' }}>
    <p>
      Method decorators are applied to the <strong>Property Descriptor</strong>{' '}
      for the method, allowing you to <strong>observe</strong>,{' '}
      <strong>modify</strong>, or <strong>replace</strong> a{' '}
      <strong>method definition</strong>.
    </p>
    <h5>Example: Memoization</h5>
    <p>
      Memoization is an <strong>optimization technique</strong> where the{' '}
      <strong>result</strong> of a function is
      <strong>cached</strong>, so if the same <strong>inputs</strong> are passed
      again, the <strong>cached result</strong> is returned instead of recalculating
      it, <strong>improving performance</strong> by avoiding
      <strong>redundant computations</strong>.
    </p>
    <h6>Decorator Factory:</h6>
  </div>
</div>

```typescript
function Memoize() {
  return function (
    target: any,
    propertyKey: string,
    descriptor: PropertyDescriptor
  ) {
    // ... (implementation will be explained next)
  };
}
```

<div style={{ padding: '5px' }}>
  <p>
    This is a decorator factory, which is a function that returns the actual
    decorator. Think of it as a way to customize our decorator before applying
    it. The decorator function it returns is the real workhorse, taking three
    important parameters:
  </p>
  <table>
    <thead>
      <tr>
        <th>Parameter</th>
        <th>Description</th>
      </tr>
    </thead>
    <tbody>
      <tr>
        <td>
          <strong>target</strong>
        </td>
        <td>
          The prototype of the class (for instance methods) or the constructor
          (for static methods).
        </td>
      </tr>
      <tr>
        <td>
          <strong>propertyKey</strong>
        </td>
        <td>The name of the method being decorated.</td>
      </tr>
      <tr>
        <td>
          <strong>descriptor</strong>
        </td>
        <td>An object that describes the method being decorated.</td>
      </tr>
    </tbody>
  </table>
</div>

<div style={{ padding: '5px' }}>
  <h6>Memoization Logic:</h6>
</div>

```typescript
const originalMethod = descriptor.value;
const cache = new Map<string, any>();

descriptor.value = function (...args: any[]) {
  const key = JSON.stringify(args);
  if (cache.has(key)) {
    return cache.get(key);
  }
  const result = originalMethod.apply(this, args);
  cache.set(key, result);
  return result;
};

return descriptor;
```

<div style={{ padding: '5px' }}>
  <p>
    This is the <strong>heart</strong> of our memoization technique. We begin by{' '}
    <strong>storing</strong> the original method and{' '}
    <strong>setting up a cache</strong> using a Map. Then, we{' '}
    <strong>replace</strong> the original method with a{' '}
    <strong>new function</strong> that does some clever work. It{' '}
    <strong>turns</strong> the arguments into a <strong>string key</strong>,{' '}
    <strong>checks</strong> if we've seen these arguments before in our cache,
    and if so, <strong>returns</strong> the stored result. If it's a{' '}
    <strong>new set</strong> of arguments, it <strong>calls</strong> the
    original method, <strong>saves</strong> the result for future use, and{' '}
    <strong>returns</strong> it. This way, we{' '}
    <strong>avoid unnecessary recalculations</strong>. Lastly, we
    <strong>return</strong> the modified descriptor, <strong>
      completing
    </strong> our memoization process.
  </p>
</div>

<div style={{ padding: '5px' }}>
  <h5>Usage of the Memoize Decorator:</h5>
</div>

```typescript
class Calculator {
  @Memoize()
  fibonacci(n: number): number {
    if (n <= 1) return n;
    return this.fibonacci(n - 1) + this.fibonacci(n - 2);
  }
}

const calculator = new Calculator();
console.log(calculator.fibonacci(10)); // 55
console.log(calculator.fibonacci(10)); // 55 (cached result)
```

<div style={{ marginTop: '20px' }}>
  <h3>Property Decorators</h3>
  <div style={{ padding: '5px' }}>
    <p>
      <strong>Property decorators</strong> are <strong>powerful tools</strong>{' '}
      that let you <strong>enhance</strong> or <strong>alter</strong> how a
      class property behaves. They act like <strong>watchful guardians</strong>,
      allowing you to
      <strong>observe</strong> when a property is <strong>accessed</strong> or <strong>
        changed
      </strong>, and even <strong>modify</strong> its behavior. This can be incredibly{' '}
      <strong>useful</strong> for adding <strong>validation</strong>, <strong>
        logging
      </strong>, or
      <strong>transforming data</strong> automatically whenever the property is
      used.
    </p>
    <h5>Example: Validation</h5>
  </div>
</div>

```typescript
function MinLength(length: number) {
  return function (target: any, propertyKey: string) {
    let value: string;
    const getter = function () {
      return value;
    };
    const setter = function (newVal: string) {
      if (newVal.length < length) {
        throw new Error(
          `${propertyKey} must be at least ${length} characters long.`
        );
      }
      value = newVal;
    };
    Object.defineProperty(target, propertyKey, {
      get: getter,
      set: setter,
    });
  };
}
```

<div style={{ padding: '5px' }}>
  <p>
    This <strong>MinLength</strong> decorator ensures that the{' '}
    <strong>username</strong> property always has a
    <strong>minimum length</strong>, throwing an <strong>error</strong> if the
    requirement is not met.
  </p>
  <h5>Usage of the MinLength Decorator:</h5>
</div>

```typescript
class User {
  @MinLength(5)
  username: string;
}

const user = new User();
user.username = 'John'; // Error: username must be at least 5 characters long.
```

<div style={{ marginTop: '20px' }}>
  <h3>Conclusion</h3>
  <div style={{ padding: '5px' }}>
    <p>
      <strong>TypeScript decorators</strong> are a{' '}
      <strong>powerful feature</strong> that can significantly enhance your
      code's <strong>functionality</strong> and <strong>readability</strong>.
      From implementing
      <strong>design patterns</strong> like <strong>Singleton</strong> to adding{' '}
      <strong>validation</strong> and <strong>memoization</strong>, decorators offer
      a <strong>clean</strong> and <strong>reusable way</strong> to <strong>
        modify
      </strong> or <strong>extend</strong> your
      <strong>classes</strong> and <strong>methods</strong>. While they're still
      an <strong>experimental feature</strong>, their potential to{' '}
      <strong>simplify complex logic</strong> and promote{' '}
      <strong>better code organization</strong> is immense. As you've seen in
      these examples, decorators can be incredibly <strong>versatile</strong>,
      allowing you to write more <strong>expressive</strong> and{' '}
      <strong>maintainable TypeScript code</strong>.
    </p>
  </div>
</div>


Deep Dive into TypeScript Decorators


<div style={{ marginTop: "20px" }}>
  <h3>Introduction</h3>
  <div style={{ padding: "5px" }}>
    <p>
      <strong>Amazon S3 events</strong> let you automatically respond to changes
      in your storage, like when files are <strong>uploaded</strong>,{" "}
      <strong>deleted</strong>, or <strong>modified</strong>. By linking S3 with{" "}
      <strong>AWS Lambda</strong>, you can create <strong>workflows</strong>{" "}
      that instantly react to these events without needing to manage servers.
      This makes it easy to automate tasks like{" "}
      <strong>processing uploads</strong>, <strong>resizing images</strong>, or{" "}
      <strong>updating records</strong> in real-time. In this post, we’ll dive
      into how you can handle <strong>S3 events</strong> with{" "}
      <strong>AWS Lambda</strong> in a simple and efficient way.
    </p>
  </div>
</div>

<div style={{ marginTop: "20px" }}>
  <h3>Implementing S3 Event Handling with AWS Lambda</h3>
  <div style={{ padding: "5px" }}>
    <p>
      To understand what exactly happens when an <strong>S3 event</strong> is
      triggered, this post will guide you through the implementation of the
      architecture below. By examining a practical example, we’ll explore how{" "}
      <strong>S3 events</strong> are processed and handled using{" "}
      <strong>AWS Lambda</strong>, providing you with a clear, hands-on
      understanding of the entire <strong>workflow</strong>.
    </p>
  </div>
  <img
    src="/images/posts/s3-events-lambda/1.png"
    style={{ marginTop: "5px" }}
  />
  <div style={{ padding: "5px" }}>
    <p>
      The idea is to upload an image to an <strong>S3 bucket</strong> named{" "}
      <strong>original-images</strong>. This upload will trigger an{" "}
      <strong>S3 event</strong> that is intercepted by a{" "}
      <strong>Lambda function</strong> written in <strong>Node.js</strong>. The
      Lambda function processes the incoming event, retrieves the uploaded
      image, and generates a <strong>thumbnail version</strong>. This automated
      process efficiently handles <strong>image resizing</strong> and stores the
      resulting <strong>thumbnails</strong> in a designated{" "}
      <strong>thumbnails bucket</strong>.
    </p>
  </div>
</div>

<div style={{ marginTop: "20px" }}>
  <h3>Provision resources using CloudFormation</h3>
  <div style={{ padding: "5px" }}>
    <p>
      <strong>AWS CloudFormation</strong> is a service that helps you automate
      the setup and management of AWS resources using code. Instead of manually
      configuring individual services, you define them in a{" "}
      <strong>JSON</strong> or <strong>YAML template</strong>. CloudFormation
      then <strong>provisions</strong>, <strong>updates</strong>, or{" "}
      <strong>deletes</strong> your resources in a consistent and repeatable
      way, making <strong>infrastructure management</strong> more efficient and
      reliable.
    </p>
  </div>
</div>

<div style={{ marginTop: "20px" }}>
  <h4>Create the CloudFormation template</h4>
  <div style={{ padding: "5px" }}>
    <p>
      For me, I prefer to use <strong>YAML</strong>, but feel free to use{" "}
      <strong>JSON</strong> if you want. Let's start by creating a file called{" "}
      <strong>deployment.yaml</strong>.
    </p>
  </div>
</div>

```yaml
#deployment.yaml
AWSTemplateFormatVersion: "2010-09-09"
Description: "CloudFormation template to provision resources"
```

<div style={{ padding: "5px" }}>
  <p>
    These two lines in the template specify the version of the template format
    being used and provide a description of the template's purpose.
  </p>
  <p style={{ marginTop: "2px" }}>
    Now, let's add a <strong>Resources</strong> section to the template and
    start by creating the logical resource for the first{" "}
    <strong>S3 bucket</strong>. This bucket will store the{" "}
    <strong>original images</strong> and trigger the{" "}
    <strong>file creation event</strong>.
  </p>
  <p style={{ marginTop: "2px" }}>
    <strong>Note</strong>: Resources inside the CloudFormation template are
    called <strong>logical resources</strong>. When you create a CloudFormation
    stack based on the template, a <strong>physical resource</strong> is created
    for each logical resource.
  </p>
</div>

```yaml
Resources:
  # First S3 Bucket
  OriginalImagesBucket:
    Type: "AWS::S3::Bucket"
    Properties:
      BucketName: "mz-original-images-07"
      NotificationConfiguration:
        LambdaConfigurations:
          - Event: "s3:ObjectCreated:*"
            Function: !GetAtt MzFunction.Arn
```

<div style={{ padding: "5px" }}>
  <p>
    This snippet defines the <strong>mz-original-images-07</strong> in the
    template:
  </p>
  <ul>
    <li>
      <strong>OriginalImagesBucket</strong>: This is the logical name for the S3
      bucket resource.
    </li>
    <li>
      <strong>Type</strong>: Specifies the AWS resource type, which is{" "}
      <strong>AWS::S3::Bucket</strong> in this case.
    </li>
    <li>
      <strong>Properties</strong>: Contains the configuration details for the S3
      bucket.
    </li>
    <li>
      <strong>BucketName</strong>: The name of the bucket, which is{" "}
      <strong>mz-original-images-07</strong>.
    </li>
    <li>
      <strong>NotificationConfiguration</strong>: Configures the bucket to send
      notifications.
    </li>
    <li>
      <strong>LambdaConfigurations</strong>: Specifies the Lambda function to be
      triggered when an object is created in the bucket.
    </li>
    <li>
      <strong>Event</strong>: The type of event that triggers the Lambda
      function, in this case, <strong>s3:ObjectCreated:*</strong>.
    </li>
    <li>
      <strong>Function</strong>: Refers to the ARN of the Lambda function, which
      is specified as <strong>!GetAtt MzFunction.Arn</strong>. The Lambda
      function is named <strong>MzFunction</strong>.
    </li>
  </ul>
</div>

<div style={{ padding: "5px" }}>
  <h5>Defines the mz-thumbnails-07 bucket</h5>
</div>

```yaml
# Second S3 Bucket
ThumbnailsBucket:
  Type: "AWS::S3::Bucket"
  Properties:
    BucketName: "mz-thumbnails-07"
```

<div style={{ padding: "5px" }}>
  <p style={{ marginTop: "2px" }}>
    {" "}
    <strong>Note:</strong> S3 bucket names need to be <strong>
      globally unique
    </strong>, which is why I added <strong>mz</strong> and <strong>07</strong> to
    the names.
  </p>
</div>

<div style={{ padding: "5px" }}>
  <h5>Defines the MzFunction</h5>
  <p style={{ marginTop: "2px" }}>
    Before defining the <strong>Mz Lambda function</strong> itself, we first
    need to define the <strong>execution role</strong> that the function will
    use in order to react to the <strong>creation event</strong>, process the
    file, and then store it in the other bucket. The Lambda function needs the
    proper <strong>permissions</strong> to perform all of these actions.
  </p>
</div>

```yaml
# IAM Role for Lambda
LambdaExecutionRole:
  Type: "AWS::IAM::Role"
  Properties:
    RoleName: "LambdaExecutionRole"
    AssumeRolePolicyDocument:
      Version: "2012-10-17"
      Statement:
        - Effect: "Allow"
          Principal:
            Service: "lambda.amazonaws.com"
          Action: "sts:AssumeRole"
    ManagedPolicyArns:
      - "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
      - "arn:aws:iam::aws:policy/AmazonS3FullAccess"
```

<div style={{ padding: "5px" }}>
  <p>
    This defines an IAM role called <strong>LambdaExecutionRole</strong> that
    allows AWS Lambda to assume the role and execute functions. It grants the
    Lambda function basic execution permissions and full access to S3 through
    managed policies.
  </p>

  <p>
    Now, let's jump into defining our MZFunction itself
  </p>
</div>

```yaml
# Lambda Function
MzFunction:
  Type: "AWS::Lambda::Function"
  Properties:
    FunctionName: "mz-function"
    Handler: "index.handler"
    Role: !GetAtt LambdaExecutionRole.Arn
    Runtime: "nodejs18.x"
    Timeout: 30
    MemorySize: 128
    Code:
      ZipFile: |
        exports.handler = async (event) => {
          console.log("Empty Lambda", event);
          return true;
        }
```

<div style={{ padding: "5px" }}>
  <p>
    This <strong>AWS CloudFormation</strong> code above creates our{" "}
    <strong>Lambda function</strong> "mz-function" using{" "}
    <strong>Node.js 18.x</strong>. It runs the <strong>index.handler</strong>{" "}
    function with a <strong>30-second</strong>
    timeout, <strong>128 MB</strong> of memory. About the function code, I've added
    a simple
    <strong>console.log()</strong> for now, and we'll update it later.
  </p>
  <p>
    The last thing we have to do is to give the S3 for our originals image the
    permission to triggered the Lambda function.
  </p>
</div>

```yaml
# S3 Bucket Notification for Lambda Trigger
LambdaInvokePermission:
  Type: "AWS::Lambda::Permission"
  Properties:
    FunctionName: !GetAtt MzFunction.Arn
    Action: "lambda:InvokeFunction"
    Principal: "s3.amazonaws.com"
    SourceArn: "arn:aws:s3:::mz-original-images-07"
```

<div style={{ padding: "5px" }}>
  <p>
    The above AWS CloudFormation code grants S3 permission to invoke the Lambda
    function MzFunction when an event occurs. It allows the S3 bucket
    mz-original-images-07 to trigger the function. The Principal is set to S3,
    specifying the source of the event.
  </p>
</div>

<div style={{ marginTop: "20px" }}>
  <h3>Deploy the CloudFormation Template</h3>
  <div style={{ padding: "5px" }}>
    <p>
      To deploy a CloudFormation template via the AWS Management Console, follow
      these steps:
    </p>
    <table style={{ width: "100%", borderCollapse: "collapse" }}>
      <thead>
        <tr>
          <th style={{ border: "1px solid #ddd", padding: "8px" }}>Step</th>
          <th style={{ border: "1px solid #ddd", padding: "8px" }}>Action</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <td style={{ border: "1px solid #ddd", padding: "8px" }}>1</td>
          <td style={{ border: "1px solid #ddd", padding: "8px" }}>
            <strong>Sign in to AWS Console:</strong> Open the AWS Management
            Console and navigate to the CloudFormation service.
          </td>
        </tr>
        <tr>
          <td style={{ border: "1px solid #ddd", padding: "8px" }}>2</td>
          <td style={{ border: "1px solid #ddd", padding: "8px" }}>
            <strong>Create Stack:</strong> Click on "Create stack" and select
            "With new resources (standard)."
          </td>
        </tr>
        <tr>
          <td style={{ border: "1px solid #ddd", padding: "8px" }}>3</td>
          <td style={{ border: "1px solid #ddd", padding: "8px" }}>
            <strong>Upload Template:</strong> Choose "Upload a template file"
            and select our <strong>deployment.yaml</strong> template.
          </td>
        </tr>
        <tr>
          <td style={{ border: "1px solid #ddd", padding: "8px" }}>4</td>
          <td style={{ border: "1px solid #ddd", padding: "8px" }}>
            <strong>Configure Stack:</strong> Enter a stack name, configure
            stack options, and set parameters as required.
          </td>
        </tr>
        <tr>
          <td style={{ border: "1px solid #ddd", padding: "8px" }}>5</td>
          <td style={{ border: "1px solid #ddd", padding: "8px" }}>
            <strong>Review and Create:</strong> Review your settings, and click
            "Create stack" to deploy the template. AWS will start provisioning
            the resources defined in your template.
          </td>
        </tr>
      </tbody>
    </table>
    <p style={{ marginTop: "5px" }}>
      To visualize the template's logical resources, simply click the{" "}
      <strong>View in Application Composer</strong> button.
    </p>
  </div>
</div>

<img
  src="/images/posts/s3-events-lambda/2.png"
  style={{ display: "block", margin: "0 auto" }}
/>

<div style={{ padding: "5px" }}>
  <p>After a few minutes, our resources will be created successfully</p>
</div>
<img
  src="/images/posts/s3-events-lambda/3.png"
  style={{ display: "block", margin: "0 auto" }}
/>

<div style={{ marginTop: "20px" }}>
  <h3>Test if the Lambda function is triggered when an image is uploaded</h3>
  <div style={{ padding: "5px" }}>
    <p>
      To view the console.log output of our Lambda function, follow these steps:
    </p>
    <table style={{ width: "100%", borderCollapse: "collapse" }}>
      <thead>
        <tr>
          <th style={{ border: "1px solid #ddd", padding: "8px" }}>Step</th>
          <th style={{ border: "1px solid #ddd", padding: "8px" }}>Action</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <td style={{ border: "1px solid #ddd", padding: "8px" }}>1</td>
          <td style={{ border: "1px solid #ddd", padding: "8px" }}>
            Upload an image to the S3 Originals Images bucket.
          </td>
        </tr>
        <tr>
          <td style={{ border: "1px solid #ddd", padding: "8px" }}>2</td>
          <td style={{ border: "1px solid #ddd", padding: "8px" }}>
            Go to the Lambda service.
          </td>
        </tr>
        <tr>
          <td style={{ border: "1px solid #ddd", padding: "8px" }}>3</td>
          <td style={{ border: "1px solid #ddd", padding: "8px" }}>
            Select our Lambda function: Choose the Lambda function whose logs we
            want to view.
          </td>
        </tr>
        <tr>
          <td style={{ border: "1px solid #ddd", padding: "8px" }}>4</td>
          <td style={{ border: "1px solid #ddd", padding: "8px" }}>
            Go to the Monitoring tab:
            <ul style={{ margin: "0", padding: "0 0 0 20px" }}>
              <li>
                Click on the Monitoring tab for the selected Lambda function.
              </li>
              <li>
                Under the Logs section, click on View logs in CloudWatch. This
                will open the CloudWatch Logs console.
              </li>
            </ul>
          </td>
        </tr>
        <tr>
          <td style={{ border: "1px solid #ddd", padding: "8px" }}>5</td>
          <td style={{ border: "1px solid #ddd", padding: "8px" }}>
            View logs in CloudWatch:
            <ul style={{ margin: "0", padding: "0 0 0 20px" }}>
              <li>
                In the CloudWatch Logs console, look for the log group with the
                name /aws/lambda/mz-function.
              </li>
              <li>Click on the log group to view the log streams.</li>
              <li>
                Select the most recent log stream to see the output of our
                console.log statements.
              </li>
            </ul>
          </td>
        </tr>
      </tbody>
    </table>
  </div>
</div>

<img
  src="/images/posts/s3-events-lambda/4.png"
  style={{ display: "block", margin: "0 auto", marginTop: "5px" }}
/>
<center>
  <small>mz-function console.log output</small>
</center>

<img
  src="/images/posts/s3-events-lambda/5.png"
  style={{ display: "block", margin: "0 auto", marginTop: "5px" }}
/>

<center>
  <small>mz-function configs</small>
</center>

<div style={{ marginTop: "20px" }}>
  <h3>Add the lambda function code</h3>
  <div style={{ padding: "5px" }}>
  <p>To add the Lambda code, follow these steps in the AWS Management Console:</p>
<ol>
  <li>Navigate to the <strong>Code</strong> tab for your Lambda function.</li>
  <li>Enter the following code:</li>
</ol>

  </div>
</div>

```js
const s3 = new (require("aws-sdk").S3)();

// Define the source and destination buckets
const SOURCE_BUCKET = "mz-original-images-07"; // Original images bucket
const THUMBNAILS_BUCKET = "mz-thumbnails-07"; // Thumbnails bucket

exports.handler = async (event) => {
  // Extract S3 object information from the event
  const record = event.Records[0];
  const bucket = record.s3.bucket.name;
  const key = decodeURIComponent(record.s3.object.key.replace(/\+/g, " "));

  // Check if the bucket is the source bucket
  if (bucket !== SOURCE_BUCKET) {
    console.log("Not an image upload event for the source bucket.");
    return;
  }

  try {
    // Get the image from S3
    const params = {
      Bucket: bucket,
      Key: key,
    };
    const { Body } = await s3.getObject(params).promise();

    // Process the image (Here, we just re-upload the original image for demonstration)
    // To resize, you would typically need a library, but we'll skip resizing for simplicity

    // Define the destination key
    const thumbnailKey = `thumbnails/${key}`;

    // Upload the image to the thumbnails bucket
    const uploadParams = {
      Bucket: THUMBNAILS_BUCKET,
      Key: thumbnailKey,
      Body: Body,
      ContentType: "image/jpeg",
    };
    await s3.putObject(uploadParams).promise();

    console.log(`Image uploaded to ${THUMBNAILS_BUCKET}/${thumbnailKey}`);
  } catch (error) {
    console.error("Error processing the image:", error);
    throw error;
  }
};
```

<div style={{ marginTop: "20px" }}>
  <div style={{ padding: "5px" }}>
    <p>
      the code is about extracting the image from an S3 bucket, checks if it's
      from the source bucket, and then re-uploads it to a different S3 bucket.
    </p>
    <p>
      <strong>Note 1</strong> : The image is simply copied without resizing for
      demonstration purposes.
    </p>
    <p>
      <strong>Note 2</strong> : If you encounter the error
      "Runtime.ImportModuleError: Error: Cannot find module 'aws-sdk'" whenever
      your Lambda function is triggered, try downgrading your Node.js runtime
      from the AWS Management Console. This issue is known to occur with the
      Node.js 18 runtime, and switching to an earlier version may resolve it.
    </p>
    <p>
      <strong>Finally</strong> : click the Deploy button to apply the changes to
      our Lambda code.
    </p>
  </div>
</div>

<div style={{ marginTop: "20px" }}>
  <h3>Conclusion</h3>
  <div style={{ padding: "5px" }}>
    <p>
      <p>
        In this blog, we demonstrated how to deploy a CloudFormation template
        and configure an AWS Lambda function to respond to S3 events. The Lambda
        function is triggered when an image is uploaded to an S3 bucket,
        allowing for automated processing or handling of the image. This setup
        illustrates the basics of integrating S3 with Lambda to streamline
        workflows in a serverless environment.
      </p>
    </p>
  </div>
</div>


Automating S3 Events with AWS Lambda


<div style={{ marginTop: "20px" }}>
  <h3>Introduction</h3>
  <div style={{ padding: "5px" }}>
    JavaScript Generators are a powerful feature that can give you the ability
    to create functions that can be paused and then resumed. Besides handling
    asynchronous operations, they provide an elegant way to create iterables. In
    this post, we will learn about generators and use it with real life examples
    suitable for frontend ( React) as well as backend ( Node. js) development.
  </div>
</div>

<div style={{ marginTop: "20px" }}>
  <h3>What are Generators</h3>
  <div style={{ padding: "5px" }}>
   <p>
  JavaScript <strong>generators</strong> are a special type of function that can be <strong>paused</strong> and <strong>resumed</strong>, making them useful for handling complex tasks. Unlike normal functions, which run until completion, generators allow you to control when and how much of the function executes at a time. They are created using the <code>function*</code> syntax.
</p>

<p>
  The key feature of generators is the <strong>yield</strong> keyword, which is
  used to return a value and pause the function’s execution. When you call the
  generator again, it resumes from where it left off, continuing until the next{" "}
  <strong>yield</strong> statement or the end of the function.
</p>

<p>
  Generators also work as <strong>iterators</strong>, meaning they can be used
  in loops or with <code>.next()</code> calls to retrieve values step by step.
  Each time you invoke <code>.next()</code>, it provides the next value and
  pauses again.
</p>

<p>
  A common use case for generators is managing large datasets or infinite
  sequences, as they allow you to process data incrementally, which saves
  memory. Another popular use is in asynchronous programming, where generators
  can make code flow more readable by pausing at certain steps.
</p>

<p>
  Here's a quick example of a generator:
</p>
 </div>
</div>

```javascript
function* numberGenerator() {
  yield 1;
  yield 2;
  yield 3;
}

const gen = numberGenerator();
console.log(gen.next().value); // 1
console.log(gen.next().value); // 2
console.log(gen.next().value); // 3
```

![generators-js](/images/posts/js-generators/1.png)

<div style={{ marginTop: "20px" }}>
  <h3>Frontend Use Case: Infinite Scrolling in React</h3>
  <div style={{ padding: "5px" }}>
    One practical application of generators in frontend development is
    implementing infinite scrolling. Let's create a React component that uses a
    generator to fetch data in chunks as the user scrolls
  </div>
</div>

```javascript
import React, { useState, useEffect } from "react";

function* dataFetcher() {
  let page = 1;
  while (true) {
    yield fetch(`https://api.example.com/data?page=${page}`).then((response) =>
      response.json()
    );
    page++;
  }
}

function InfiniteScrollList() {
  const [items, setItems] = useState([]);
  const [fetcher, setFetcher] = useState(null);

  useEffect(() => {
    setFetcher(dataFetcher());
  }, []);

  const loadMore = async () => {
    if (fetcher) {
      const { value } = await fetcher.next();
      const newItems = await value;
      setItems((prevItems) => [...prevItems, ...newItems]);
    }
  };

  return (
    <div>
      <ul>
        {items.map((item) => (
          <li key={item.id}>{item.name}</li>
        ))}
      </ul>
      <button onClick={loadMore}>Load More</button>
    </div>
  );
}

export default InfiniteScrollList;
```

<div style={{ marginTop: "20px" }}>
  <div style={{ padding: "5px" }}>
    In this example, the <strong>dataFetcher</strong> generator yields promises
    that fetch data from an API. The React component uses this generator to load
    more items when the user clicks a button. You could easily extend this to
    trigger on scroll events for true infinite scrolling.
  </div>
</div>

<div style={{ marginTop: "20px" }}>
  <h3>Backend Use Case: Streaming Large Datasets with Node.js</h3>
  <div style={{ padding: "5px" }}>
    Generators are incredibly useful in backend development, especially when
    dealing with large datasets. Let's create a Node.js script that uses a
    generator to stream a large dataset from a database and process it in
    chunks.
  </div>
</div>

```javascript
const { MongoClient } = require("mongodb");

async function* largeDatasetStreamer(collection, batchSize = 1000) {
  let skip = 0;
  while (true) {
    const batch = await collection.find().skip(skip).limit(batchSize).toArray();
    if (batch.length === 0) break;
    yield batch;
    skip += batchSize;
  }
}

async function processLargeDataset() {
  const client = new MongoClient("mongodb://localhost:27017");
  await client.connect();

  const db = client.db("mydatabase");
  const collection = db.collection("largedata");

  const streamer = largeDatasetStreamer(collection);

  for await (const batch of streamer) {
    for (const item of batch) {
      // Process each item
      console.log(item);
    }
  }

  await client.close();
}

processLargeDataset().catch(console.error);
```

<div style={{ marginTop: "20px" }}>
  <div style={{ padding: "5px" }}>
    In this Node.js example, we create a generator function largeDatasetStreamer
    that yields batches of data from a MongoDB collection. The
    processLargeDataset function uses this generator to iterate over the entire
    dataset in manageable chunks, allowing for efficient processing of large
    amounts of data without loading everything into memory at once.
  </div>
</div>

<div style={{ marginTop: "20px" }}>
  <h3>Conclusion</h3>
  <div style={{ padding: "5px" }}>
    JavaScript generators offer a unique and powerful way to handle asynchronous
    operations and manage complex flows in both frontend and backend
    development. Whether you're implementing infinite scrolling in a React
    application or processing large datasets in Node.js, generators can help you
    write more efficient and maintainable code. By leveraging generators, you
    can create more responsive user interfaces and build scalable backend
    systems that can handle large amounts of data with ease. As you've seen in
    the examples above, generators provide a clean and intuitive API for working
    with asynchronous and iterative processes, making them a valuable tool in
    any JavaScript developer's toolkit.
  </div>
</div>


The art of Javascript Generators


<div style={{ marginTop: "20px" }}>
  <h3>Introduction</h3>
  <div style={{ padding: "5px" }}>
    AWS enforces encryption for objects stored within an S3 bucket to ensure
    data security at rest. This practice is known as server-side encryption
    (SSE), which comprises three types:
    <ul>
      <li>
        <b>Server-Side Encryption with Customer Provided Keys (SSE-C)</b>
      </li>
      <li>
        <b>Server-Side Encryption with S3-Managed Keys (SSE-S3)</b>
      </li>
      <li>
        <b>Server-Side Encryption with KMS Keys (SSE-KMS)</b>
      </li>
    </ul>
  </div>
</div>

<div style={{ marginTop: "20px" }}>
  <h3>Server-Side Encryption with Customer Provided Keys</h3>
  <div style={{ padding: "5px" }}>
    <p>
      With this method, you take control of key management by supplying S3 with
      the data in plain text and an encryption key. S3 then uses this key to
      encrypt the data, storing both the ciphertext of the encrypted object and
      a hash of the key. The key itself is securely destroyed.
    </p>
    <img src="/images/posts/s3-sse/1.png" style={{ marginTop: "5px" }} />
    <p style={{ marginTop: "5px" }}>
      to decrypt the object, you must provide S3 with the key. S3 will verify if
      the provided key is correct and, if successful, decrypt the object,
      returning the plaintextand descards the key. This approach empowers users
      to handle the encryption keys, adding an extra layer of control over data
      security.
    </p>
  </div>
</div>

<div style={{ marginTop: "20px" }}>
  <h3>Server-Side Encryption with S3-Managed Keys</h3>
  <div style={{ padding: "5px" }}>
    <p>
      With this method S3 handles the both processes : managing the keys and
      encrypt/decrypt the data and it is the default method that S3 uses. You
      just need to provide the plaintext data and S3 will creates, manages and
      rotates the key so you have zero control of the key.
    </p>
    <img src="/images/posts/s3-sse/2.png" style={{ marginTop: "5px" }} />
    <p style={{ marginTop: "5px" }}>
      SSE-S3 uses the Advanced Encryption Standard (AES) algorithm with a key
      size of 256 bits. AES-256 is a widely adopted encryption standard known
      for its strength and security. When you retrieve an object from an SSE-S3
      encrypted bucket, Amazon S3 automatically decrypts the object before
      returning it to the requester. As a user, you do not need to perform any
      additional steps to decrypt the data.
    </p>
    <p style={{ marginTop: "5px" }}>
      There is no additional cost for using SSE-S3. It is included as part of
      the standard Amazon S3 service offerings but there is a significant
      drawback associated with this method, primarily related to Role
      Separation. Since everyone with access to S3 possesses the ability to read
      encrypted data, it raises concerns about data privacy and security. In
      many scenarios, you may want to restrict access to encrypted data to
      specific individuals or roles within your organization.
    </p>
  </div>
</div>

<div style={{ marginTop: "20px" }}>
  <h3>Server-Side Encryption with KMS Keys</h3>
  <div style={{ padding: "5px" }}>
    <p>
      With this approach, another AWS service involved in the process is AWS Key
      Management Service (KMS). KMS is responsible for managing keys with role
      separation. In order to use a specific key generated by KMS, you need to
      have permissions and these controlled by IAM (Identity and Access
      Management).
    </p>
    <img src="/images/posts/s3-sse/3.png" style={{ marginTop: "5px" }} />
    <p style={{ marginTop: "5px" }}>
      In this case, S3 will solely handle the encryption and decryption
      processes, while KMS will manage the keys with isolated permissions that
      are configurable by you. Each time S3 requests a key from KMS to encrypt
      specific data, KMS will provide both a Plaintext key and a Ciphertext
      version of the encryption key (DEK). S3 will use the plaintext version of
      the key for encryption, then discard it and store the ciphertext version
      along with the encrypted data for later use in the decryption process.
    </p>
  </div>
</div>

<div style={{ marginTop: "20px" }}>
  <h3>Conclusion</h3>
  <div style={{ padding: "5px" }}>
    <p>
      S3 server-side encryption offers three methods: SSE-C, SSE-S3, and
      SSE-KMS, each tailored to diverse encryption needs. SSE-C empowers users
      with key management control, while SSE-S3 streamlines encryption with
      automated key management. However, SSE-S3 poses concerns about role
      separation, as it grants universal access to encrypted data. In contrast,
      SSE-KMS enhances security by leveraging AWS KMS for key management,
      ensuring customizable permissions and role separation.
    </p>
  </div>
</div>


S3 Server-Side Encryption


<div style={{ marginTop: "20px" }}>
  <h3>Introduction</h3>
  <div style={{ padding: "5px" }}>
    IAM (Identity and Access Management) stands out as a global AWS service,
    distributed across all regions, serving as the foundational service for
    managing access to resources within your AWS account.
    <p>
      When users seek to perform actions on resources, such as creating an EC2
      instance, specific <b>permissions</b> are required. IAM empowers the
      creation of users and enables precise control over <b>permissions</b> and
      <b>authorizations</b> through IAM <b>policies</b>.
    </p>
    <p>
      An IAM <b>User</b> is designated as an identity, and the system introduces
      two additional identity types: <b>Roles</b> and <b>Groups</b>. These
      identities provide enhanced flexibility in access management, offering a
      comprehensive approach to permissions and organizational control.
    </p>
    <img src="/images/posts/iam/3.jpg" alt="IAM Image" />
  </div>
</div>

<div style={{ marginTop: "20px" }}>
  <h3>IAM Policies</h3>
  <div style={{ padding: "5px" }}>
    <p>
      An IAM Policy serves as a mechanism to either grant or deny permissions to
      identities within the AWS environment. These policies, presented in JSON
      or YAML format, are linked to specific identities.
    </p>
    <p>
      Once a policy is attached, the associated identity operates strictly
      within the boundaries defined by the policy. Here's a simple example of an
      IAM Policy:
    </p>
      <img src="/images/posts/iam/7.png" alt="IAM Image" />
   <p>
      In this example, we create an IAM Policy that grants create, stop, and
      terminate EC2 instances:
    </p>
    <ul>
      <li>
        <b>Version</b>: This specifies the version of the IAM policy language.
        In this case, it's set to "2012-10-17," which is a standard version.
      </li>
      <li>
        <b>Statement</b>: This is an array of policy statements. Each statement
        in the array represents a set of permissions.
      </li>
      <li>
        <b>Effect</b>: This indicates that the specified actions are allowed.
        You could also use "Effect": "Deny" to explicitly deny the actions.
      </li>
      <li>
        <b>Action</b>: This lists the AWS actions that are allowed. In this
        case, the policy allows the actions of running, stopping, and
        terminating EC2 instances.
      </li>
      <li>
        <b>Resource</b>: This specifies the resource to which the actions apply.
        Replace your-region with the AWS region code where your instances
        reside, and replace your-account-id with your AWS account ID. The
        instance/* at the end indicates that these permissions apply to all EC2
        instances (* is a wildcard for any instance ID).
      </li>
    </ul>
    <p style={{ marginTop: "10px" }}>
      In this example, we are granting permissions specifically for handling EC2
      instances. It's important to note that, in IAM, the order of evaluation
      follows a specific sequence: Explicit Deny, Explicit Allow, and finally,
      Implicit Deny. This means that if an action is not explicitly allowed for
      EC2 instances, it will implicitly be denied. Careful attention to this
      order is crucial for crafting policies to ensure that only intended
      actions are permitted, and that security measures are in place.
    </p>
  </div>
  <h5>Evaluate multiple policies</h5>
  <div style={{ padding: "5px" }}>
    when multiple policies are attached to an identity (user, group, or role),
    the policies are evaluated based on a specific order of precedence. The
    priority follows these rules:
    <p>
      <b>Explicit Deny Overrides Explicit Allow:</b>
      <div>
        If there is an explicit Deny statement in any of the attached policies,
        it takes precedence over any Allow statements. This means that if any
        policy explicitly denies an action, it will override an explicit allow
        from another policy.
      </div>
    </p>
    <p>
      <b>Policy Evaluation Order:</b>
      <div>
        Policies are evaluated based on the order in which they are attached to
        the identity. Policies attached later in the list take precedence over
        policies attached earlier.
      </div>
    </p>
    <p>
      <b>User Policy vs. Group Policy:</b>
      <div>
        If a user is a member of multiple groups and both the user's policy and
        group policies grant permissions for the same action, the user's policy
        takes precedence.
      </div>
    </p>
    <p>
      <b>Explicit Allow in a Policy:</b>
      <div>
        If a policy explicitly allows an action, and no other policy explicitly
        denies the same action, the action is allowed.
      </div>
    </p>
    <p>
      <b>Implicit Deny:</b>
      <div>
        If there are no policies attached that explicitly allow an action, and
        there is no explicit allow at the identity level (e.g., on the user or
        group), then the action is implicitly denied. It's crucial to design
        policies carefully to avoid unintended consequences. Explicitly denying
        specific actions can be a useful security practice, but it requires
        careful consideration to prevent accidental denial of necessary
        permissions. Additionally, understanding the order of evaluation helps
        in crafting policies that align with the desired access control model.
      </div>
    </p>
  </div>
  <h5>Types of policies</h5>
  <div style={{ padding: "5px" }}>
    In AWS Identity and Access Management (IAM), there are mainly two types of
    policies:
    <div>
      <h6>Managed Policies</h6>
      <p>
        <b>AWS Managed Policies:</b> These are policies created and managed by
        AWS. You can attach these policies to multiple users, groups, or roles
        in your AWS account.
      </p>
      <img src="/images/posts/iam/1.png" alt="IAM Image" />
      <p>
        <b>Customer Managed Policies:</b> These are policies that you create and
        manage. You can attach them to multiple users, groups, or roles in your
        AWS account.
      </p>
      <img src="/images/posts/iam/2.png" alt="IAM Image" />
       <h6 style={{marginTop : "5px"}}>Inline Policies</h6>
       <p>
         These are policies that you create and manage directly within a user, group, or role. Unlike managed policies, inline policies are embedded directly into the resource's definition and are deleted when the resource is deleted.
Policies, whether managed or inline, define the permissions for identities (users, groups, or roles) in AWS, specifying what actions are allowed or denied on what resources.
       </p>
    </div>

  </div>
</div>
<div style={{ marginTop: "20px" }}>
  <h3>IAM Users</h3>
  <div style={{ padding: "5px" }}>
  <p>
   An IAM user is the principal identity in IAM. Each user possesses their own set of credentials, including access keys. Users serve as a means of long-term access to AWS resources, enabling human access to the AWS Cloud through the console.
   Additionally, they can be utilized by applications to access resources via the access that you can create after the user creation process.
   </p>
   <div>
   <h5>Example of creating a user to access AWS S3 via the CLI:</h5>
   <h6 style={{marginTop : "5px"}}>1: Create a user through the AWS console</h6>
    <img src="/images/posts/iam/4.png" alt="IAM Image" />
    <h6 style={{marginTop : "5px"}}>2: Create Access Keys for the user</h6>
    <img src="/images/posts/iam/5.png" alt="IAM Image" />
    <p style={{marginTop : "5px"}}>
    Then install the AWS CLI on your machine if it's not already installed and open
     a Terminal or Command Prompt and type the following command and press Enter 
      <center><img src="/images/posts/iam/8.png" alt="IAM Image" /></center>
      <p style={{marginTop : "5px"}}>
      You'll be prompted to enter your AWS Access Key ID and Secret Access Key 
      and specify the AWS region you want to use as the default. Enter the code for your preferred region (e.g., us-east-1).
      </p>
      <p style={{marginTop : "5px"}}>
      After entering all information, the AWS CLI will display a message indicating 
      that the configuration was successful.
      </p>
       <p style={{marginTop : "5px"}}>
        <center>
        Now we are ready to try it:
        <img src="/images/posts/iam/6.png" alt="IAM Image" />
        it works 🥳🥳
        </center>
      </p>
      <p>
      There is a hard limit of 5000 users per account. To overcome this limitation, IAM roles come into play as another type of identity.
      </p>
    </p>
   </div>
  </div>
  <div style={{ marginTop: "20px" }}>
  <h3>IAM Roles</h3>
  <div style={{ padding: "5px" }}>
   <p>
   IAM roles in AWS (Identity and Access Management) are a way to grant permissions to entities that you trust. Roles are not associated with a specific user or group; instead, they are assumed by trusted entities,
    such as AWS services, applications, or users from other AWS accounts.
   </p>
   <h5>Key points about IAM Roles</h5>
   <p style={{ marginTop: "5px" }}>
   <b>Purpose:</b> Roles are used to delegate permissions, allowing
    entities to perform actions in your AWS account on your behalf.
   </p>
   <p style={{ marginTop: "5px" }}>
   <b>Temporary Credentials:</b> When a role is assumed, it provides temporary security 
   credentials, including an access key, secret key, and session token. 
   These credentials are used to make AWS service API requests.
   </p>
   <p style={{ marginTop: "5px" }}>
    <b>Cross-Account Access:</b> Roles are commonly used for cross-account access,
     enabling users or services in one AWS account to access resources in another account.
   </p>
   <p style={{ marginTop: "5px" }}>
   <b>Service Roles:</b> Roles can be assumed by AWS services. For example, 
   an EC2 instance can assume a role to access other AWS services securely.
   </p>
   <p style={{ marginTop: "5px" }}>
   <b>Trust Policy:</b> A role has a trust policy that specifies who can assume the
    role. It defines the trusted entities (AWS accounts, IAM users, or AWS services) 
    that are allowed to assume the role.
   </p>
   <p style={{ marginTop: "5px" }}>
   <b>Permissions Policy:</b> The role also has an associated permissions policy that
    defines the actions allowed or denied when someone assumes the role.
   </p>
   <p  style={{ marginTop: "5px" }}>
   Here's an example of a trust policy for a role:
   </p>
   <img src="/images/posts/iam/9.png" alt="IAM Image" />
   <center>In this example, the role can be assumed by the EC2 service.</center>
  </div>
   <p  style={{ marginTop: "5px" }}>
   Roles provide a secure and flexible way to manage access to AWS resources, especially in scenarios involving third-party applications, cross-account access, and temporary permissions.
   </p>
  </div>
</div>
 <div style={{ marginTop: "20px" }}>
  <h3>IAM Groups</h3>
  <div style={{ padding: "5px" }}>
  <p>
  A group is a collection of IAM users. Groups let you specify permissions for multiple users, which can make it easier to manage access in your AWS environment. 
  Instead of attaching policies to individual users, you can attach policies to a group, and users added to that group inherit the permissions.
  </p>
  <h5>Key points about IAM Groups</h5>
   <p style={{ marginTop: "5px" }}>
    <b>User Management:</b> users can be added to multiple groups, and each group can have different policies attached, allowing for flexible access management.
    When a user is added to a group, they inherit the permissions assigned to that group.
   </p>
   <p style={{ marginTop: "5px" }}>
   <b>Policy Attachment:</b> policies are attached directly to groups, and users in the group inherit those policies.
   This helps in applying a set of permissions consistently to multiple users.
   </p>
   <p style={{ marginTop: "5px" }}>
   <b>Simplifies Access Management:</b> group membership simplifies access management, especially when dealing with a large number of users who require similar permissions.
   </p>
   <p style={{ marginTop: "5px" }}>
   <b>No Credentials:</b> a group itself doesn't have credentials or access keys so you can't login to AWS account via a specific group. 
   Users within the group have their own credentials.
   </p>
   <p style={{ marginTop: "5px" }}>
   <b>Policy Inheritance:</b> users inherit the permissions of all the groups they belong to, as well as any individual policies that are attached directly to the user.
IAM groups are a fundamental component of IAM that helps in organizing and managing user permissions efficiently in AWS.
   </p>
  </div>
  </div>

<div style={{ marginTop: "20px" }}>
  <h3>Conclusion</h3>
  <div style={{ padding: "5px" }}>
    IAM is a vital AWS service, globally distributed, managing resource access.
    Users, roles, and groups, controlled by IAM policies, offer flexible access
    management. IAM policies, in JSON or YAML, grant or deny permissions,
    ensuring secure resource handling. IAM groups streamline access management,
    providing a cohesive structure for permissions.
  </div>
</div>


AWS IAM Service


<div style={{ marginTop: "20px" }}>
  <h3>Introduction</h3>
  <div style={{ padding: "5px" }}>
    AWS is a global cloud platform powered by a global infrastructure that
    consists of a network of data centers and facilities around the world.
    <p>
      AWS provides cloud computing services to millions of customers, and its
      global infrastructure is designed to ensure high availability,
      reliability, and low-latency access to its services.
    </p>
  </div>
</div>

<div style={{ marginTop: "20px" }}>
  <h3>Components</h3>
  <div style={{ padding: "5px" }}>
    There are four key components that form the backbone of the AWS global
    infrastructure:
    <ul>
      <li>
        <strong>Regions</strong>
      </li>
      <li>
        <strong>Availability Zones</strong>
      </li>
      <li>
        <strong>Edge Locations</strong>
      </li>
      <li>
        <strong>Local Zones</strong>
      </li>
    </ul>
  </div>
</div>

<div style={{ marginTop: "20px" }}>
  <h3>Regions</h3>
  <div style={{ padding: "5px" }}>
    A region is a geographical area where AWS has established a presence to host its cloud computing services. Each AWS region is essentially a separate and independent data center infrastructure, strategically located around the world.
  </div>

<div style={{ padding: "5px" }}>
  With this separation, cloud architects have the ability to design highly
  available applications that operate across multiple regions. In the event of a
  specific region experiencing downtime, the applications remain functional
  because the operations in the unaffected region are not impacted by the issues
  in the downed region. This approach enhances resilience and ensures continuous
  service availability, reinforcing the robustness of the overall architecture.
</div>

![Maple](/images/posts/aws-global/2.png)

<div style={{ padding: "5px" }}>
  Regions exist around the world, providing businesses with the opportunity to
  deploy their applications and solutions in the closest geographical proximity
  to their customers. This strategic placement facilitates low-latency access,
  contributing to improved application performance and a more responsive user
  experience. By leveraging AWS regions, businesses can optimize the
  geographical distribution of their resources, ensuring efficient delivery of
  services and minimizing the time it takes for data to travel between end-users
  and the deployed infrastructure.
</div>

  <div style={{ padding: "5px" }}>
    AWS presently operates 33 regions globally, each distinguished by a unique name and code. 
    Additionally, they have plans to launch an additional 6 regions in the near future.

  </div>

  <div style={{ padding: "5px" }}>
    When interacting with an AWS service, your actions are confined to a designated region.
     For example, if you wish to provision an EC2 instance (a virtual machine), 
     you must select a specific region, as an EC2 instance in the us-east-1 (N. Virginia) 
     region is distinct from an EC2 instance in the eu-central-1 (Frankfurt) region.
    Some services do not necessitate the selection of a specific region
     these are referred to as global services.
  </div>
</div>

<div style={{ marginTop: "20px" }}>
  <h3>Avalabilty Zone</h3>
  <div style={{ padding: "5px" }}>
    <p>
      Each AWS region comprises a minimum of three Availability Zones (AZs) and
      may extend to include four, five, or six AZs. These Availability Zones are
      meticulously separated, ensuring that the failure of a specific AZ does
      not impact the functionality of the others. They are interconnected
      through high-bandwidth, low-latency networking, further enhancing the
      overall reliability and resilience of the infrastructure.
    </p>
    <p
      style={{
        marginTop: "5px",
      }}
    >
      AWS does not explicitly disclose the composition of an Availability Zone
      (AZ), but it can be either a single data center or multiple data centers.
    </p>
    <p style={{ marginTop: "5px" }}>
      In the context of the earlier example, when provisioning an EC2 instance,
      we must also select a specific Availability Zone (AZ) inside the region
      since an EC2 instance operates exclusively within a single AZ.
    </p>
    <p style={{ marginTop: "5px" }}>
      Additionally, since EC2 is a private service, it operates only from a
      Virtual Private Cloud (VPC). A VPC represents a secluded cloud network
      segmented into subnets, which can be classified as either private or
      public. Explicit configuration is essential to facilitate communication
      between the internet and any resource provisioned within the VPC.
    </p>
    <div style={{ marginTop: "5px" }}>
      <h5>The final architecture:</h5>
      <img src="/images/posts/aws-global/3.png" />
    </div>
  </div>
</div>

<div style={{ marginTop: "20px" }}>
  <h3>Edge locations</h3>
  <div style={{ padding: "5px" }}>
    <p>
      Edge locations are AWS data centers designed to deliver services with the
      lowest latency possible. Amazon has dozens of these data centers spread
      across the world. They’re closer to users than Regions or Availability
      Zones, often in major cities, so responses can be fast and snappy.
    </p>
    <div style={{ marginTop: "5px" }}>
      Edge locations are used for a number of AWS features like{" "}
      <b>CloudFront</b> and
      <b>AWS Global Accelerator</b>
      <p style={{ marginTop: "5px" }}>
        <h5>CloudFront</h5>
        CloudFront is the most commonly discussed use of edge locations. It’s a content
        delivery network that caches content in edge locations. Content can be served
        directly from the cache, so it gets to users faster. CloudFront is often
        used to serve static assets, speed up websites, and stream video.
      </p>
      <img src="/images/posts/aws-global/4.png" />
      <p>
        Notice that there are two types of <b>Edge Locations</b> local edges and
        regional edges. This mechanism operates by first checking the local
        edge. If there's a hit, the request is immediately returned. Otherwise,
        it checks the closest regional edges. If cached data exists, it is
        returned to the user. If not, the regional edge retrieves the data from
        the original source such as S3 and this done via <b>CloudFront</b>,
        returns it, and caches it for future requests.
      </p>
      <p style={{ marginTop: "5px" }}>
        <h5>AWS Global Accelerator</h5>
        AWS Global Accelerator allows you to route requests for key resources through
        Amazon’s global network—even if the request is going halfway round the world.
        The request is initially routed to the closest edge location and then travels
        through Amazon’s network—often with lower latency and higher throughput than
        the public internet.
      </p>
      <img src="/images/posts/aws-global/6.png" />
      <p style={{ marginTop: "5px" }}>
        Note: you can’t run your workloads directly in edge locations, they’re
        only used by Amazon’s managed services.
      </p>
    </div>
  </div>
</div>

<div style={{ marginTop: "20px" }}>
  <h3>Local zones</h3>
  <div style={{ padding: "5px" }}>
    <p style={{ padding: "5px" }}>
      Local Zones are an infrastructure deployment concept from Amazon Web
      Services (AWS) designed to bring AWS services closer to a specific
      geographic area, providing users with low-latency access to resources and
      applications.
    </p>
    <p style={{ padding: "5px" }}>
      Local Zones are an extension of the main AWS Regions and are ideal for
      latency-sensitive workloads or applications that require real-time
      processing.
    </p>
    <p style={{ padding: "5px" }}>
      These zones are often situated in large metropolitan areas and provide a
      selection of AWS services, including Elastic Compute Cloud (EC2),
      Relational Database Service (RDS), and Amazon Elastic Kubernetes Service
      (EKS), among others. By deploying resources in a Local Zone, customers can
      reduce latency for end-users in that particular location, improving
      application performance and responsiveness.
    </p>
    <img src="/images/posts/aws-global/5.png" />
    In the example above, we observe the provisioning of an EC2 instance within
    a local zone. Notably, the VPC demonstrates the capability to extend
    seamlessly from the <b>region</b> into the <b>local zone</b>. This
    integration showcases the flexibility and continuity of the Virtual Private
    Cloud (VPC) architecture, allowing resources to span across both the AZ and
    its associated local zone
  </div>
</div>

<div style={{ marginTop: "20px" }}>
  <h3>Conclusion</h3>
  <div style={{ padding: "5px" }}>
    <p>
      AWS Global Infrastructure is a robust cloud platform with key
      components—Regions, Availability Zones, Edge Locations, and Local Zones.
      These components ensure high availability, resilience, and low-latency
      access to cloud services worldwide.
    </p>
    <p style={{ marginTop: "5px" }}>
      In conclusion, AWS's infrastructure provides a flexible and responsive
      foundation, empowering businesses with efficient and reliable cloud
      solutions globally.
    </p>
  </div>
</div>


AWS Global Infrastructure


### Introduction

**Proxies** and **Load Balancers** are two essential technologies for businesses,
whether large-scale enterprises or small startups, seeking to optimize their network
infrastructure and enhance security.

### Proxies

A proxy server, often simply referred to as a proxy, acts as an intermediary between a client device and a target server and there are two main types of proxies **Forward Proxy** and **Reverse Proxy**.

#### Forward Proxy

A forward proxy is used as an intermediary between the client and a web server, it evaluates the request, takes any needed actions, and routes the request to the destination.
![Maple](/images/posts/prox-balncers/fp1.png)

The main reason of using a forward proxy is **the Anonymity and Privacy** because it allows users to browse the internet with a degree of anonymity. By routing their traffic through a proxy server, their IP address is hidden, which can be beneficial for those seeking to protect their online identity and privacy. However there are some other reasons why forward proxy is used :

##### Security

Forward proxy can inspect incoming and outgoing traffic for malicious content, which enhances security. It can scan for malware, viruses, and other threats in real-time. Additionally, forward proxy can enforce security policies, such as blocking access to known malicious websites.

##### Caching

Forward proxy can cache frequently requested web content. This caching capability accelerates access to frequently accessed resources and reduces the load on the target servers, which can lead to faster loading times for websites and reduced bandwidth consumption.

##### Bandwidth Optimization

Forward proxy can compress content before delivering it to clients, reducing bandwidth usage and improving load times. This is particularly useful in environments where bandwidth is limited or expensive.

#### Reverse Proxy

A **Reverse Proxy** is a server or gateway that sits between client devices (such as web browsers, mobile apps, or other user agents) and backend servers. It acts as an intermediary, receiving incoming client requests and forwarding them to the appropriate backend servers. Unlike a **Forward Proxy**, which is used to retrieve resources on behalf of clients, a reverse proxy is primarily used to serve as a gateway for incoming client requests to one or more backend servers.

![Maple](/images/posts/prox-balncers/rv1.png)

Here are key functions and use cases of a reverse proxy:

##### Load Balancing

One of the primary roles of a reverse proxy is load balancing. It can distribute incoming client requests across multiple backend servers, ensuring that the load is evenly distributed and no single server becomes overwhelmed. This enhances performance, scalability, and fault tolerance.

##### Security and SSL/TLS Termination

![Maple](/images/posts/prox-balncers/sec.gif)

Reverse proxies can enhance security by acting as a barrier between the internet and internal network resources. They can hide the internal structure of a network, protect sensitive information, and mitigate certain types of attacks. In some cases, reverse proxies are used to implement web application firewalls (WAFs) to protect against common security threats. Also it can handle SSL/TLS encryption and decryption on behalf of backend servers. This offloads the resource-intensive encryption and decryption tasks from the backend servers, improving performance.

##### Caching

Reverse proxies can cache content to reduce the load on backend servers and accelerate content delivery. Cached content can be served quickly to clients without the need to request the same data from the backend servers repeatedly.

##### URL Routing

Reverse proxies can perform URL routing, directing incoming requests to different backend servers based on specific URL patterns or rules. This is useful for scenarios where different services or applications are hosted on different backend servers.

##### Logging and Monitoring

Reverse proxies often provide detailed logs and statistics about incoming and outgoing traffic, which can be invaluable for monitoring and troubleshooting network activity.

#### Load Balancer

A **Load Balancer** is a special type of a Reverse Proxy that distributes incoming network traffic across multiple servers, ensuring that no single server is overwhelmed with too much demand. The primary purpose of a load balancer is to improve the availability, scalability, and performance of applications and services.

![Maple](/images/posts/prox-balncers/b1.jpg)

However there are two types of **Load Balancer:** Layer 4 Load Balancer and Layer 7 Load Balancer they operate at different levels of the OSI model, each with its own set of features and use cases.

##### Layer 4 Load Balancer (L4)

###### Layer of Operation

L4 load balancers operate at the transport layer (Layer 4) of the OSI model. They make routing decisions based on network-level information, such as IP addresses and port numbers.

###### Load Balancing Method

L4 load balancers primarily use IP and port information to distribute traffic. They do not inspect the content of the application data (e.g., HTTP headers or URL paths).

###### Performance

L4 load balancers are typically faster and have lower overhead compared to L7 load balancers because they do not need to inspect application data.

###### Limited Content Awareness

L4 load balancers lack visibility into the application data, making them less suitable for application-specific routing or optimizations.

###### Use Cases

- L4 load balancers are well-suited for TCP and UDP load balancing.
- They are commonly used for simple traffic distribution, such as distributing network traffic across multiple web servers.
- L4 load balancers are often used for non-HTTP applications, such as database connections or custom protocols.

##### Layer 7 Load Balancer (L7)

###### Layer of Operation

L7 load balancers operate at the application layer (Layer 7) of the OSI model. They have a deeper understanding of application protocols and can make routing decisions based on application-specific data.

###### Load Balancing Method

L7 load balancers can perform content-based routing, making decisions based on HTTP headers, URL paths, cookies, and other application-specific data. They are highly application-aware.

###### Content Optimization

L7 load balancers can optimize content delivery by compressing data, caching, and serving as SSL termination points.

###### Security

L7 load balancers can inspect and filter application-specific data for security purposes, such as mitigating attacks and protecting against vulnerabilities.

###### Application Awareness

L7 load balancers can provide granular control over traffic routing based on the content of the application data. This allows for features like path-based routing and URL rewriting.

###### Use Cases

- L7 load balancers are ideal for web traffic distribution, as they can perform advanced content-based routing, SSL termination, and content optimization.
- They are commonly used for web applications, services, and APIs.

In summary, the main difference between Layer 4 and Layer 7 load balancers is the level of data they operate on and the depth of application awareness.

##### Implement a basic Load Balancer using Nginx

Nginx is a powerful and flexible web server that can be configured to act as a reverse proxy and load balancer. Here's a simple Nginx configuration to set up a load balancer for distributing traffic to multiple backend servers.

- 1: Open or create an Nginx configuration file.

  - 2: In the configuration file, create an upstream block that defines your backend servers. You can specify the servers' IP addresses or hostnames along with port numbers. For example:

```bash
upstream backend_servers {
    server 192.168.1.101:80;
    server 192.168.1.102:80;
    server 192.168.1.103:80;
}
```

In this example, we've defined three backend servers with their IP addresses and port numbers.

Now let's create a server block to configure the load balancing behavior. Inside the server block, define the location and proxy_pass directives.

```bash
server {
    listen 80;
    server_name doodooti.com;

    location / {
        proxy_pass http://backend_servers;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}
```

Explain the config :

- 1: **listen 80;** specifies that Nginx should listen on port 80.
- 2: **server_name doodooti.com;** specifies the domain or hostname that Nginx should
  handle.
- 3: **location /** defines the location where the load balancing should occur.
- 4: **proxy_pass http://backend_servers;** instructs Nginx to proxy incoming requests
  to the upstream backend_servers defined earlier.

- 5: **The proxy_set_header** directives pass along some headers, such as the original
  host, client IP, and forwarded-for IP, for better logging and debugging.

The final step is to save the config and reload the Nginx server:

```bash
sudo nginx -t

sudo service nginx reload
```

With this configuration, Nginx will distribute incoming requests across the backend servers using a **round-robin** load balancing algorthim that **Nginx** uses by default.

#### Summary

In this blog, we delve into the world of Proxies and Load Balancers, exploring their key concepts and how they work, complete with practical examples. I hope you enjoy the journey of discovery.


Proxies & Load balancers


### Introduction

Hello, community 🙌🏻,

One of the key features of React is its reconciliation algorithm, which determines how the library updates the DOM in response to changes in the component state.
In this blog post, we will explore the reconciliation algorithm in depth and see how it works with examples.

### What is the reconciliation algorithm?

The reconciliation algorithm is the process React uses to update the DOM in response to changes in the component state. When a component’s state changes, React will re-render the component and its children. The reconciliation algorithm is responsible for determining what has changed in the component tree and updating the DOM accordingly.

React uses a virtual DOM (VDOM) to represent the structure of the components in memory.The virtual DOM is a lightweight in-memory representation of the DOM, and it's used by React to compare the new state of the component tree with the previous state.
If there are any differences, React will update the real DOM to match the new state.

### How does the reconciliation algorithm work?

The reconciliation algorithm starts by comparing the virtual DOM of the previous state with the virtual DOM of the new state.
If there are no differences, React will not make any changes to the real DOM. If there are differences, React will update the real DOM to match the new state.

React uses a heuristic approach to determine the most efficient way to update the real DOM. The algorithm first checks if a component has changed its type, for example, from a simple text component to a complex component with multiple children. If the component type has changed, React will unmount the old component and mount the new component.

If the component type has not changed, React will compare the properties of the old and new components. If the properties have changed, React will update the properties of the real DOM element. If the children of the component have changed, React will recursively apply the reconciliation algorithm to the children components.

React also uses a technique called reusing existing DOM nodes to make the updates more efficient. If a component has not changed its type and its properties have not changed, React will reuse the existing DOM node for that component. This allows React to avoid unnecessary DOM updates and makes the updates faster.

### Example

Let's take an example to understand the reconciliation algorithm in action:

```js
import React, { useState } from "react";

function Greeting() {
  const [name, setName] = useState("mz1");

  return (
    <>
      <h1>Hello, {name}</h1>
      <button onClick={() => setName("mz2")}>Change Name</button>
    </>
  );
}

export default Greeting;
```

In this example, when the button is clicked and the name state changes, React will run the reconciliation algorithm to update the real DOM to match the new state.

React will start by comparing the virtual DOM of the previous state with the virtual DOM of the new state.
Since the component type has not changed, React will reuse the existing DOM node for the Greeting component, then it will update the text content of the h1 element to match the new name mz2.

In this example, the reconciliation algorithm has made a single DOM update to change the text content of the h1 element.

### Optimizing the performance using Keys

In React, the use of keys is important when rendering a list of elements. Keys are a special string attribute that you can provide to elements in a list to help React keep track of which items are added, deleted, or changed.

Consider the following example of a list of names:

```js
import React from "react";

function NamesList({ names }) {
  return (
    <ul>
      {names.map((name) => (
        <li key={name}>{name}</li>
      ))}
    </ul>
  );
}

export default NamesList;
```

In this example, we're using the map function to render a list of li elements, each with a name from the names array. If the names array changes, React will use the reconciliation algorithm to update the DOM to match the new virtual DOM.

However, without the use of keys, React has no way of knowing which items have been added, deleted, or changed. As a result, React may end up unnecessarily recreating all of the li elements, even if only a single name has changed. This can lead to decreased performance, especially for long lists.

By providing a unique key to each li element, React can track which items have changed and make the minimum number of DOM updates necessary. In the example above, we're using the name itself as the key, which is a unique identifier for each item in the list.

It's important to note that the value of the key should be unique among the siblings, but doesn't have to be globally unique. It's also not recommended to use indexes as keys, because they can lead to unexpected behavior when items are reordered or deleted.

### Conclusion

The "diff algorithm" and the "reconciliation algorithm" are often used interchangeably to refer to the same thing in the context of React. However, there is a subtle difference between the two terms.

The diff algorithm refers to the process of comparing two trees of nodes, such as the virtual DOM trees in React, to determine the differences between them. The diff algorithm determines the minimum number of operations required to transform one tree into another.

The reconciliation algorithm refers to the overall process of updating the DOM in response to changes in the component state. The reconciliation algorithm includes the diff algorithm as one of its steps, but it also includes additional steps such as unmounting and mounting components, updating component properties, and handling error cases.


ReactJS: the reconciliation algorithm


### Introduction

Hello, community 🙌🏻,

JavaScript provides several ways to invoke a function, and among them are the methods bind(), call() and apply(). These methods are used to set the value of the this keyword inside a function, and they are particularly useful when working with objects and constructors. In this blog post, we'll take a closer look at each of these methods, how they work, and when to use them.

### 1. bind()

The bind() method creates a new function with a specific value for this. It does not call the function, it just creates a new function with a permanently bound this value. To use bind(), you pass in the value you want this to reference and it returns a new function with the this keyword set to the value you provided.

```js
let obj = {
  name: "John",
  sayName: function () {
    console.log(this.name);
  },
};

let boundSayName = obj.sayName.bind(obj);
boundSayName(); // Output: John
```

In this example, we have an object obj with a property name and a method sayName. We use bind() to create a new function boundSayName that has this set to obj. This means that when we call boundSayName(), this.name will be equal to "John".

### 2. call()

The call() method is used to call a function with a specific value for this. It also allows you to pass in arguments one by one. To use call(), you pass in the value you want this to reference and the arguments for the function.

```js
let obj = { name: "John" };
let obj2 = { name: "Bob" };

obj.sayName.call(obj2); // Output: Bob
```

In this example, we are calling the sayName method on the obj object and passing in obj2 as the value for this. This means that when the sayName method is called, this.name will be equal to "Bob".

### 3. apply()

The apply() method is similar to call(), but it allows you to pass in an array of arguments. To use apply(), you pass in the value you want this to reference and an array of arguments for the function.

```js
let numbers = [1, 2, 3];
console.log(Math.max.apply(null, numbers)); // Output: 3
```

In this example, we are using the apply() method to call the Math.max function and passing in an array of numbers as the arguments. The first argument is null because we don't need to set the value of this.

### Summary

In summary, bind(), call(), and apply() are powerful tools in JavaScript that allow you to set the value of this inside a function. bind() creates a new function with a permanently bound this value, call() is used to call a function with a specific value for this, and apply() is used to call a function with a specific value for this and pass in an array of arguments. Knowing when to use each of these methods can help you write more efficient and maintainable code.


this.bind(), this.apply() and this.call()


### Introduction

Hello, community 🙌🏻,

In order to build a professional web application, optimization and performance are two important things you need to care about.

There are many tips and techniques used to increase the performance of a web application, such as Debouncing and Throttling.

When it comes to debouncing and throttling, developers often confuse.

During this blog, I will go throw these two techniques in details using react.js, but it is the same principle for vanilla JavaScript or any other JavaScript framework.

### Debouncing

Before dive deep into debouncing, let's see a simple and normal example that implement a search box that allows users to search something without clicking any button.

```js
function App() {
  const handleChange = (e) => {
    console.log("api call...");
  };

  return (
    <div className="App">
      <header className="App-header">
        <p> Search </p>
        <input type="text" onChange={handleChange} />
      </header>
    </div>
  );
}
```

The issue is that handleChange is very expensive, and this is bad to the server because it will receive many HTTP requests in the same time.

![Maple](/images/posts/r-d-t/0.gif)

To solve the problem, we need to use a debounce function.

Definition and implementation of a debounce function
A debounce function is called after a specific amount of time passes since its last call.

```js
function debounce(fn, delay) {
    let timer
    return function (...args) {
      clearTimeout(timer)
      timer = setTimeout(()=>fn(...args), delay)
    }
```

The idea is to define a high-order function called debounce takes as arguments a callback function and a delay in ms, then returns a new function that sets the timer to execute the callback after the timer done.
The secret here is that every call of the function returned from the debounce function will cancel the previous timer using cleartimeout(timer) and start a new timer.
With this approach, we can be sure that the callback function will be executed just once after the time that we passed as an argument in the last call.

### Implement debounce function into our example

```js
<div className="App">
  <header className="App-header">
    <p> Search </p>
    <input type="text" onChange={debounce(handleChange, 500)} />
  </header>
</div>
```

#### Result

![Maple](/images/posts/r-d-t/1.gif)

### Throttling

Let's assume that we have an event listener in our app to track the movement of the user mouse, then send data to a backend server to do some operations based on the location of the mouse.

```js
const handleMouseMove = (e) => {
  //everytime the mouse moved this function will be invoked
  console.log("api call to do some operations...");
};
//event listener to track the movement of the mouse
window.addEventListener("mousemove", handleMouseMove);
```

If we stick with this solution, we will end up with a down backend server because it will receive a hundred of requests in short duration.

![Maple](/images/posts/r-d-t/2.gif)

1600 API calls in few seconds is very very bad 📛📛📛.
To fix this issue, we need to limit the number of API calls, and this kind of problems can be solved using a throttle function.

#### Definition and implementation of a throttle function

A throttle function is a mechanism to limit the number of calls of another function in a specific interval, any additional calls within the specified time interval will be ignored.

```js
function throttle(fn, delay) {
  let run = false;
  return function (...args) {
    if (!run) {
      fn(...args);
      run = true;
      setTimeout(() => (run = false), delay);
    }
  };
}
```

The throttle function accepts two arguments: fn, which is a function to throttle, and delay in ms of the throttling interval and returns a throttled function.

#### Implement throttle function into our example

```js
const handleMouseMove = (e) => {
  //everytime the mouse moved this function will be invoked
  console.log("api call to do some operations...");
};

//event listener to track the movement of the mouse
window.addEventListener("mousemove", throttle(handleMouseMove, 1000));
//1000ms => 1 second
```

#### Result

![Maple](/images/posts/r-d-t/3.gif)

### Conclusion

Debouncing and Throttling are two amazing techniques, they can increase the performance of your web application to another level.
Choosing one of them depends on the case.

GitHub repo: https://github.com/ridhamz/debouncing-throttling


React.js: debouncing and throttling


### Introduction

Hello community 🙌🏻,

As a JavaScript developer, knowing how to deal with asynchronous operations is an essential skill.

### 1. What are Promises 🤔 ?

Promises are a new feature introduced by ECMAScript 2015 (ES6) that is used to get the result of an asynchronous JavaScript operation like callbacks but in a fancy way.

Technically, a JavaScript promise is an object that returns a value that you hope to receive in the future.

### 2. Promise declaration syntax 🦄

![Maple](/images/posts/async-js/0.png)

<center>Promise declaration syntax</center>

Using the Promise Class we can make a promise object that takes a callback function as an argument and this function itself takes two arguments resolve and reject they are two callback functions.

resolve: get executed when the operation is done with success.

reject: get executed when the operation failed.

### 3. Simulation of real call API using Promises ⏳

![Maple](/images/posts/js-part2/1.png)

<center>Call API simulation</center>

In the example, above we implemented a simple call API simulation using the facade function setTimeout.

getUserById is a function that takes id as an argument and returns the user if exists, otherwise returns false.

Depending on the result of the getUserById function, if the user exists, we execute the resolve function and pass the user data as an argument or execute the reject function and pass an error message as an argument.

### 4. Promise object status 📉

The promise object has 3 different status depending on the asynchronous operation.

Pending if the asynchronous operation is still not yet done.

resolved if the asynchronous operation was done successfully.

rejected if the asynchronous operation failed.

### 5. Getting the Promise result ✅

Getting the result of a Promise using .then .catch syntax.

![Maple](/images/posts/async-js/2.png)

<center>.then .catch syntax</center>

The promise class provides two methods, then used to inject a function when the status is resolved and catch to inject a function when the status is rejected.

Besides that, the Promise class provides a third method called finally that allows us to run a function in both cases resolved or rejected.

![Maple](/images/posts/async-js/3.png)

<center>Finally example</center>

### 6. Promise Chaining ⛓️

As we know that every promise object has 3 methods (then, catch, finally) the thing is that every method returns a new promise.

![Maple](/images/posts/async-js/4.png)

<center>Promise Chaining Example</center>

In the example above :

- We take the result of the promise with the first method then and return a new promise that resolves a new value: res + 1 (1 + 1)
- In the second then method, we accept the result of the promise that returns from the first then and returns a new promise by resolving a new value res × 2 (2 × 2)
- The final result was printed using the third, then method.

Promise Changing is a powerful and clean approach used to manipulate the result of a promise on multiple levels. (it is the same for catch and finally)

### 7. Promise.all() and Promise.race() ⚙️

- Promise.all() : static method takes an array of Promises as an argument and returns a new Promise that resolves the resolved values from all Promises that exists in the array, but if one of these promises is rejected, then the Promise returned is also rejected with an error from the first rejected promise.

![Maple](/images/posts/async-js/5.png)

<center>Promise.all() resolve example</center>

![Maple](/images/posts/async-js/6.png)

<center>Promise.all() reject example</center>

- Promise.race() : static method takes an array of Promises as an argument and returns a new Promise depending on the first Promise that resolves or rejects from all Promises exists in the array and first means first in time not first in order.

<center>Promise.all() resolve example</center>

![Maple](/images/posts/async-js/7.png)

<center>Promise.race() example</center>

### 8. async-await 💎

async-await is a JavaScript feature introduced by ECMAScript 2016 (ES7) in the order to give us more options to deal with asynchronous operations.

async-await built on top of Promises so we can use it to get the result of a promise object also it gives us more clean code and more readable code.

To see the beauty of async-await let’s first refactor our getUserById function

![Maple](/images/posts/async-js/8.png)

<center>Refactor getUserById</center>

now let’s see how to get the result of this function using async-await syntax

![Maple](/images/posts/async-js/9.png)

<center>async-await syntax</center>

With this approach, we can just get the resolved value cause if the promise rejects we will get nothing so how to handle errors when we use async-await?

The answer is JavaScript try-catch :

![Maple](/images/posts/async-js/10.png)

<center>Catch error using try-catch</center>

try-catch is awesome but repeating it with every call of an asynchronous function is a little bit annoying, that is why we need to write a generic function that can handle any other asynchronous function.

![Maple](/images/posts/async-js/11.png)

<center>handleAsync function</center>

handleAsync takes an asynchronous callback function and returns an array containing the result and the error.

Using JavaScript destructuring, we can get the values of the result and the error.

![Maple](/images/posts/async-js/12.png)

<center>Using handleAsync</center>

we used getResult function cause in JavaScript await is allowed to use only in an async function but if you don’t like to use getResult() you can use an IIFE function (Immediately Invoked Function Expression).

![Maple](/images/posts/async-js/13.png)

<center>Using handleAsync in an IIFE function</center>

### 9. Asynchronous operations inside loops 💪🏻

By default, JavaScript loops don’t support asynchronous operations but there is a way to do asynchronous operation inside a loop.

![Maple](/images/posts/async-js/14.png)

<center>asynchronous with map</center>

We can do asynchronous operations inside a map by wrapping it with await Promise.all()


Asynchronous JavaScript:Promises and async-await


### Introduction

Hello community 🙌🏻,

This is part 2 of“ How does javascript work in the background” blog series 📒.

Today we will talk about Global memory, Local memory, Scope & Variable Scope, Higher Order functions and one of the most important and powerful javascript concept it is closure.

### 1. Global Memory 🌎

Global memory is where all global variables are stored, it is connected to the global execution context. Every variable or function declared in the global execution context will be stored in the global memory.
![Maple](/images/posts/js-part2/0.png)

In the first reading, javascript will put all variables and functions declaration in the global memory as key-value pairs, the variable count will take the initial value 0 and the function will take the code definition as a value.

### 2. Local Memory 🗺️

Every javascript function has own local memory used to store local variables.

When a function tries to get variable value it looks first in its local memory if the variable doesn’t exist then the function will be searching for it in the global memory.

### 3. Scope & Variable Scope 🔭

The scope is a mechanism used by the language in order to handle the visibility and the access of variables.

There are many types of scope like lexical scope, dynamic scope… however javascript use the lexical scope called also static scope.

The main idea of lexical scope is that child’s function can access any variable declared in the parent context. We can say that functions are lexically bound to the execution context of their parents, but the parent can’t access any variable declared in a child context.

![Maple](/images/posts/js-part2/1.png)

In the example above the function calculate can access to a and b but if we try to access the variable sum outside calculate we will get an error ❌ ‘ReferenceError: sum is not defined’ and the same thing if we try to access b outside printSomme.

Another thing you need to know about javascript scope is that inside the same function if there is a variable declared with let or const keyword inside a block {} can’t be accessed outside this block it is called block scope however a variable declared with the keyword var can’t have the block scope.

### 4. Higher-Order functions ⚡

the basic definition is that any function takes another function as an argument or/and returns a function as result called ‘Higher Order function’.

![Maple](/images/posts/js-part2/2.png)

<center>Higher Order Function Simple Example</center>

![Maple](/images/posts/js-part2/3.png)

<center>Higher Order Function Advanced Example</center>

### 5. Closure 🏆

Closure is when a function remembers the variables around it, even when that function is executed in a different context.

![Maple](/images/posts/js-part2/4.png)

<center>getSum execution context</center>

In the example above when the getSum function gets invoked in line 13 javascript will push the function into the call stack and a brain new execution context will be created a and b will be stored in the local memory as key-value pairs.

When the execution of getSum() is done, the brain new execution context and the local memory will be deleted, and the function will be removed from the call stack and the result of that function will be stored in the calc variable.

![Maple](/images/posts/js-part2/5.png)

<center>End of getSum execution context</center>

Now calc holds the calculate function that returns the sum of a and b but these two variables don’t exist
either in the global memory, neither in the local memory of the calculate function, and the getSum function
is deleted with its local memory. The question is how the calculate function will get the value of these
two variables a and b?

![Maple](/images/posts/js-part2/6.png)

<center>calc execution context</center>

To answer this question we need to know that every function returned, as a result, will take all variables that it needs when will get invoked, these variables are stored in small memory attached to the function called backpack.

![Maple](/images/posts/js-part2/7.png)

<center>function backpack</center>

So the idea is to get any variable value the function checks the backpack first if this variable doesn’t exist in the backpack the function will check the local memory if also doesn’t exist then the function will look at the global memory.

We call all this process Closure.


How does javascript work in the background (part 2)


### Introduction

Hello, community 🙌🏻,

Right now javascript is one of the most popular programming languages in the world.

In this blog, I will share the most important concepts about javascript that you have to know as a web developer.

### 1. Thread of execution and Global execution context 💡

As we know javascript is a single thread language this is mean that every javascript code executes line by line using Thread of execution, and this process happens inside a container called the global execution context.

![Maple](/static/images/posts/js-part1/1.png)

But this global execution context can contain lines of code and also functions, so how can javascript handle this situation?

The answer is that every function in javascript has its own execution context, when a function gets invoked javascript will create a brain new execution context then the thread of execution will switch from the global execution context to the brain new execution context and starts running function’s code line by line, Javascript manages all the execution contexts with something called Call Stack.

### 2. Call Stack (Last In First Out) 🧮

By default, the call stack contains the global execution context, when a function gets invoked javascript push it inside the call stack so the thread of execution switch from global to this new execution context, and when the function has done javascript will remove it from the call stack then the thread of execution go back to the global.
![Maple](/images/posts/js-part1/2.png)

In this example when the thread of execution comes to line 7 javascript will figure out that this is not a simple line of code, but it is a function call, so it will push this function into the call stack and create a brain new execution context so the thread of execution switch from the global to this new execution context and go throw the code of SayHello() line by line.

![Maple](/images/posts/js-part1/3.png)

### 3. CallBacks, CallBack Queue and Event Loop ➿

Javascript is a single thread language it can’t handle many operations in the same time but what makes it very powerful is its asynchronous nature this is mean that javascript when executing an asynchronous operation didn’t wait for the result of this operation because waiting the result of an asynchronous code will make it a slow language (example get data from database). So the idea is to run any asynchronous code without waiting for the result and jump to the next line when the result is ready javascript gets it using callBack functions. To simplify, a callback function is a normal function gets executed when an asynchronous operation is done in order to get or manipulate the result.

But the question now is where javascript stores these callback functions and how can it manage their order?. The answer is CallBack Queue basically it is a simple queue (first in last out) used just to store callback functions until the asynchronous operation is done.

![Maple](/images/posts/js-part1/4.png)

In this example, we have an asynchronous function called getUserDataFromApi this function will take 1 second to get the data I used setTimout function to simulate the real HTTP request, to get the result I passed a callBack function called ShowUserData takes userdata as an argument and console log it.

Let’s what will happen when the thread of execution runs the function getUserdataFromApi() at line 13 :

![Maple](/images/posts/js-part1/5.png)

When the thread arrives at line 8 getUserdataFromApi() will be pushed into the Call Stack, a brain new execution context will be created, and the callBack function ShowuserData will be pushed into CallBack Queue.

javascript will not wait 1 sec for the result, but it will remove getUserdataFromApi() from the call stack and go back to global to execute the next line and console log the number 2.

After 1sec the result will be done and the call stack will be empty then javascript will remove ShowUserData() and push it into the call stack and a brain new execution context will be created.

![Maple](/images/posts/js-part1/6.png)

There is an important information you need to know that any function in the callback queue will be pushed into the call stack just when this call stack is empty, but the question is how javascript will know if the stack is empty or not?

Here we introduce the Event Loop, which is responsible for checking if the stack is empty or not. If the call stack is not empty even after 2 sec, the ShowUserData will wait for more time.

The output of our example will be :

1

2

user data


doodooti - Your new favorite spot !

Recent Posts