Add annotations to an image

Written byPhuoc Nguyen

Created

15 Nov, 2023

#Selecting a portion of an image

In our previous post, we learned how to select a portion of an image and implement the crop functionality. But now, we're going to use that selection for a different purpose: adding annotations.

To do this, we created a custom hook called `userClickToSelect`. With this hook, users can click on an element and drag their mouse to select a portion of it.

Using the `useClickToSelect` hook is easy-peasy.

const [clickToSelectRef, dx, dy, startX, startY, selection] = useClickToSelect();

The hook returns an array of six items that we can use to create a smooth and real-time movement of an element as users move the mouse. The first item is the reference of the target element, which can be attached using the `ref` attribute. The next two items, `dx` and `dy`, represent the width and height of the element, while the `startX` and `startY` values indicate how far the mouse has moved relative to the element's container.

We can use these four values to set the corresponding styles of the element. The last item, selection, lets us check whether users are dragging the target element. We can compare the `selection` value with `Selection.Dragging` to see if the selected portion is shown only when users are moving the mouse.

Here's a code snippet to remind you of what we've done:

tsx

<div className="annotator__cropper" ref={clickToSelectRef}>
{
    selection === Selection.Dragging && (
        <div
            className="annotator__selection"
            style={{
                transform: `translate(${startX}px, ${startY}px)`,
                width: `${dx}px`,
                height: `${dy}px`,
            }}
        />
    )
}
</div>

Once the user releases the mouse, we create an element to represent the selected portion. This element is only displayed after the user releases the mouse, so we check the `selection` value against the `Selection.Selected` state. We also compare the `dx` and `dy` values to ensure that both the width and height of the selected portion are greater than zero.

The selected portion has the same size as the area the user dragged around, so we use the `dx` and `dy` values to set the width and height of the portion.

tsx

{
    dx > 0 && dy > 0 && selection === Selection.Selected && (
        <div
            className="annotator__cropping"
            style={{
                width: `${dx}px`,
                height: `${dy}px`,
            }}
        >
            ...
        </div>
    )
}

To make the selection draggable and resizable, we use additional hooks:

const [draggableRef, dragX, dragY] = useDraggable();
const [resizableRef] = useResizable();

The `useDraggable` hook makes an element draggable, while the `useResizable` hook makes an element resizable. But if we want an element to have both functionalities, we can use the `mergeRefs` function. This function merges different references together, creating a single reference with all the functionalities provided by the different refs passed to the `mergeRefs` function.

const croppingMergedRef = mergeRefs([draggableRef, resizableRef]);

In this example, the `croppingMergedRef` is what enables the target element to be draggable and resizable. We simply pass it to the `ref` attribute of the selected portion. It's important to note that we indicate the dimension for the portion by passing the `dx` and `dy` values to the width and height styles, respectively. However, we haven't positioned it yet.

Fortunately, the `useDraggable` hook also returns `dragX` and `dragY` values that are in sync with the top-left corner of the element that the user is dragging. We use these values with the `translate` function to define the position of the portion area and pass it to the `transform` property.

Here's a sample code to show how we do it:

tsx

<div
    ref={croppingMergedRef}
    className="annotator__cropping"
    style={{
        transform: `translate(${dragX}px, ${dragY}px)`,
    }}
>
    ...
</div>

Take a look at the demo below where you can easily click and drag to select a portion of the image. Once you release the mouse, you can then drag or resize the selected portion within the image container. Give it a try!

Photo by Ivana Cajina on Unsplash

#Displaying a form to add annotations

Once the selected portion is displayed, we'll need to provide users with a popover to add an annotation. To achieve this, we'll create a new CSS class to position the popover. The popover will be positioned absolutely within the container, so we'll set the `position` to `absolute`. We'll also set the `top` and `left` properties to zero, so it will appear initially at the top-left corner.

css

.annotator__popover {
    position: absolute;
    top: 0;
    left: 0;
}

To make the user experience better, we can add an arrow near the top-left corner to show which part of the image users are interacting with. Instead of creating a new element for the arrow, we can use the `::before` element.

First, we set its `height` and `width` to 0.5rem and its `position` to `absolute`. Then we use the `border-top` and `border-left` properties to make a triangle shape. By rotating the element 45 degrees with `transform: rotate(45deg)`, we can make it look like an arrow pointing towards the selected part of the image.

Finally, we use `transform: translate(50%, -50%)` to adjust the position of the arrow so that it points directly at the selected part.

css

.annotator__popover::before {
    background-color: inherit;
    border-top: 1px solid rgb(203 213 225);
    border-left: 1px solid rgb(203 213 225);

    content: '';
    position: absolute;
    left: 0.5rem;
    top: 0;
    transform: translate(50%, -50%) rotate(45deg);

    height: 0.5rem;
    width: 0.5rem;
}

To figure out where the popover should appear, we use a simple logic. First, we check if both the `dx` and `dy` values are greater than zero, and if the current `selection` state is `Selection.Selected`. These checks make sure that users have released the mouse and generated a portion.

Then, we use the `translate` function to set the popover position. We pass the `dragX` value to the first parameter of the function, which aligns the popover with the selected portion horizontally. The second parameter is the sum of the `dragY` and `dy`, which pushes the popover to the bottom of the selected portion. We add a small value (8) to create some space between the popover and the selected portion.

Below is some sample code that shows how we position the popover:

tsx

{
    dx > 0 && dy > 0 && selection === Selection.Selected && (
        <div
            className="annotator__popover"
            style={{
                transform: `translate(${dragX}px, ${dragY + dy + 8}px)`,
            }}
        >
            ...
        </div>
    )
}

Now, the popover is positioned correctly right after the user releases the mouse. However, we're not quite done yet. When you resize the height of the selected portion using the handler at the bottom, the popover position doesn't update accordingly.

To fix this issue, we'll use a custom hook called `useWatchSize` that watches the width and height of an element. We won't go into the implementation of this hook here, but you can find all the details at this link.

This hook returns an array of three items. The first item is the reference of the target element that we want to track the dimensions of, while the remaining items are the element's up-to-date dimensions.

const [watchSizeRef, w, h] = useWatchSize();
const croppingMergedRef = mergeRefs([draggableRef, resizableRef, watchSizeRef]);

In this example, we use the `mergeRefs` function to combine three refs returned by three hooks. This ensures that the `h` value returned by the `useWatchSize` hook is in sync with the height of the selected portion. We then use this `h` value to determine the position of the popover. Here's an updated version of the sample code that shows the popover.

tsx

{
    dx > 0 && dy > 0 && selection === Selection.Selected && (
        <div
            className="annotator__popover"
            style={{
                transform: `translate(${dragX}px, ${dragY + h + 8}px)`,
            }}
        >
            ...
        </div>
    )
}

We've solved the problem of positioning the popover, but we've run into another issue. When we try to interact with the popover by clicking on it, it simply closes. This is because we're handling the `mousedown` event of the container for click-and-drag functionality.

To avoid this issue, we need to move the popover out of the container for the image. Instead, we can place the image container inside another `div` element and position the popover within it.

Here's how we can modify the markup to ensure that the popover interaction doesn't interfere with the image.

tsx

<div className="annotator">
    <div className="annotator__cropper">...</div>

    {
        dx > 0 && dy > 0 && selection === Selection.Selected && (
            <div className="annotator__popover">...</div>
        )
    }
</div>

In this example, we've put both the popover and cropper elements inside a new `div` element. This new div element has `annotator` CSS, and its `position` style is set to `relative`. This way, the popover is positioned correctly within the new `div` element.

css

.annotator {
    position: relative;
}

Take a look at the demo below. Try resizing the selected position, and you'll see that the popover position changes accordingly.

#Understanding the data model

Before we dive into the annotations, let's talk about the data model used to represent them. Each annotation can be represented by a TypeScript type.

type Annotation = {
    id: string;
    x: number;
    y: number;
    h: number;
    w: number;
    text: string;
}

The `Annotation` data model is a TypeScript interface that describes the annotation object's structure. It has five properties: `id`, `x`, `y`, `h`, `w`, and `text`.

The `id` property is crucial because it helps us identify each annotation uniquely. We can use it to edit or delete an annotation. When a user adds an annotation in our app, we generate a unique `id` using the `generateString` function. But in a real-world scenario, we might send a request to the server and get the `id` from the backend.

The remaining properties define where on an image the annotation appears and what text should be displayed with it. The `x` and `y` properties represent the top-left corner of the annotated portion, while `h` and `w` represent the height and width of the annotated area. The `text` property contains any text associated with the annotation.

Using this data model, we can easily store and retrieve annotations from a backend database or local storage. We can also use it to display existing annotations when loading an image and allow users to add new ones.

We keep track of the list of annotations using the `annotations` state variable, which starts out empty.

const [annotations, setAnnotations] = React.useState([]);

When a user adds a new annotation, we create a new object with properties `x`, `y`, `h`, `w`, and `text`. We use the `generateString` function to give the annotation a unique identifier, which we add as the `id` property.

To update the `annotations` array, we create a new array with all the previous annotations and add this newly created annotation object. Then, we use the `setAnnotations` function to set the updated array back to the state. This triggers React to re-render our component with the updated annotations.

Here's the simple code we use to handle adding annotations:

const handleAddAnnotation = (text) => {
    setAnnotations((annotations) => ([
        ...annotations,
        {
            id: generateId(6),
            text,
            x: dragX,
            y: dragY,
            w,
            h,
        },
    ]));
};

When a user deletes an annotation, we filter it out from our current list using its unique `id` property, and then reset the list using the `setAnnotations` function. By managing our annotations in this way, we can easily display existing annotations when loading an image. Plus, users can add or delete annotations with ease.

#Showing annotations

To show the annotations that already exist, we use a loop to go through the `annotations` array. For each annotation, we use a separate component called `Annotation` to display it on the image. We pass the `annotation` object as a prop to this component.

The code snippet below demonstrates how we use the `map` function to loop through the annotations array and display each annotation as an instance of the `Annotation` component:

tsx

{
    annotations.map((annotation) => (
        <Annotation annotation={annotation} key={annotation.id} />
    ))
}

To keep track of each component's identity and quickly update or re-render components if there are any changes, we use the `key` prop in React.

By keeping all the rendering logic inside a separate component, we make the code more modular and easier to read and maintain. If we need to change how annotations are displayed or add new functionality, we only need to modify this one component instead of changing multiple places where annotations are used.

The `Annotation` component is responsible for rendering an individual annotation item. It takes in an `annotation` prop that contains all the necessary information for displaying the annotation on the image.

Inside the component, we first set the position of the annotation using the `transform` CSS property, which allows us to translate it by `x` pixels horizontally and `y` pixels vertically from the top-left corner of the image.

Additionally, we set the width and height of the annotation using its `w` and `h` properties, respectively. This ensures that each annotation is displayed at the correct size and position on top of the image.

Finally, we display any text associated with this annotation. The text is obtained from the `text` property. This way, we can easily customize how each annotation is displayed without having to modify multiple parts of the code.

tsx

const Annotation = ({ annotation }) => {
    return (
        <div
            className="annotator__annotation"
            style={{
                transform: `translate(${annotation.x}px, ${annotation.y}px)`,
                width: `${annotation.w}px`,
                height: `${annotation.h}px`,
            }}
        >
            {annotation.text}
        </div>
    );
};

While annotations can be helpful, displaying them all the time can be annoying, especially when there are many on an image. It clutters the interface and makes it hard to focus on individual annotations. So, we need a way to show the annotation text only when needed, like when a user hovers over or clicks on an annotation. This will make our interface more user-friendly and improve the overall experience.

To accomplish this, we'll use a custom hook called `useClickOutside` to check if users click outside of an element. We won't go into the details of the hook, but if you're interested, you can check out this post.

The hook returns an array of two items.

const [ref, isOpen] = useClickOutside();

The first thing we need is the reference of the target element, which we can attach to using the `ref` attribute. The second thing we need is a boolean value that tells us whether users have clicked inside the element or not. If they have, then we set `isOpen` to true. We can use this value to render the annotation within a popover.

tsx

const Annotation = ({ annotation }) => {
    const [ref, isOpen] = useClickOutside();

    return (
        <div
            className="annotator__annotation"
            ref={ref}
        >
            {isOpen && (
                <div
                    className="annotator__popover"
                    style={{
                        transform: `translateY(${annotation.h + 8}px)`,
                    }}
                >
                    {annotation.text}
                </div>
            )}
        </div>
    );
};

Give the final demo a try below. When the form for adding a new annotation opens, simply enter your text and press Enter. If you change your mind, just hit the Escape key. You'll notice that there's nothing overly complicated about how we implemented this feature, as you can see in the `AddNewAnnotation` component.

Once the annotation is created, just click on the corresponding section to view the text.

#Conclusion

To sum up, React is an efficient and effective tool for adding annotations to images. By using various hooks and CSS properties, we built a custom annotation tool that enables users to select, resize, and add annotations to specific parts of an image.

React's component-based architecture allowed us to break down complex functionalities into smaller, reusable components, making our code more readable and maintainable. Although there is still room for improvement, such as adding missing functionalities like editing and removing annotations, we leave these tasks for you to handle.

Overall, React provides a powerful toolkit for building rich user interfaces with advanced functionalities like image annotations. It allows for quick prototyping, easy maintenance, and robust performance while delivering a seamless user experience.

#See also

If you found this post helpful, please consider giving the repository a star on GitHub or sharing the post on your favorite social networks 😍. Your support would mean a lot to me!

Crop a portion of an image←

Take a screenshot of a portion of an element→

Questions? 🙋

Do you have any questions about front-end development? If so, feel free to create a new issue on GitHub using the button below. I'm happy to help with any topic you'd like to learn more about, even beyond what's covered in this post.

While I have a long list of upcoming topics, I'm always eager to prioritize your questions and ideas for future content. Let's learn and grow together! Sharing knowledge is the best way to elevate ourselves 🥷.

Ask me questions

Newsletter 🔔

If you're into front-end technologies and you want to see more of the content I'm creating, then you might want to consider subscribing to my newsletter.

By subscribing, you'll be the first to know about new articles, products, and exclusive promotions.

Don't worry, I won't spam you. And if you ever change your mind, you can unsubscribe at any time.

Phước Nguyễn