Click here to Skip to main content
13,146,742 members (79,740 online)
Click here to Skip to main content
Add your own
alternative version

Stats

7.7K views
236 downloads
9 bookmarked
Posted 1 Dec 2016

Web System for Smart Image Viewing: Upload, Deep Zoom, Mark Out and Annotation

, 1 Dec 2016
Rate this:
Please Sign up or sign in to vote.
Compact software for images upload, processing to tile pyramid and subsequent Deep Zoom viewing with custom mark out and annotation.

Table of Contents

 

 

Figure 1. The system structure.

Introduction

In many modern applications there are requirements for easy and efficient work with large high resolution images. Users need tools to load and view such images with seamless zooming and pan capabilities without the loading time or latency, mark out and annotate certain regions of them. It's even better to have these tools available as platform independent Web applications with zero footprints on a user's machine / device. To address this need, the technology called Deep Zoom (DZ) [1] was developed by Seadragon Software team [2] within Microsoft. DZ was shipped with Microsoft Silverlight used in Silverlight Multiscale Image control. It worked just fine in Web applications but required browser support for Silverlight [3]. Open Seadragon (OSD) project [4] provides similar capabilities framework agnostic as a completely open source. This work presents Web based software using OSD DZ technology for large images viewing.

OSD provides developer API [5] available from JavaScript (JS) to be used in Web browser. This API supports also overlay mechanism [6] for mark out. OSD overlays use "flying" HTML elements placed over selected regions in image. It seems that such an approach is not quite handy for online mark out, particularly for regions with irregular shape. So, a different technique for mark out and annotation based on direct drawing in canvas rather than HTML elements usage is implemented in this work. The structure of the system is depicted in Figure 1. Components of the presented software are listed in the following table.

 

Table. Software Components.

ComponentDescriptionTools
Image Upload ServerImage upload to file storage and start of Image Processor to make image tile pyramidnode.js, JavaScript
Upload PageA simple Web page backing by Image Upload Server providing UI for the uploadHTML
Image ProcessorComponent to create image tile pyramid out of original image.NET Core DLL (currently running in Windows only)
Image Viewing ServerSupports viewing Web applicationnode.js, JavaScript
Viewing Web ApplicationWeb application for viewing, mark out and annotation of the imageHTML, CSS, JavaScript, TypeScript

Functionality

The system works as following. When user would like to upload an image she/he browses on https://UploadServerURL:15005 and gets a very simple page for image upload. Then the user finds the required image and by pressing button Upload posts the image to Upload Server. The latter accepts uploaded file, places it to a file storage and runs Image Processor in a separate process. Image Processor dzPyramidBuilder.dll is a .NET Core DLL. Its mission is to create image tile pyramid from an original image. When the image pyramid has been created, the image is ready for DZ viewing and annotation performed by the viewing subsystem.

To view and possibly annotate the image, the user should browse on https://ViewingServerURL:15000 yielding the page shown in Figures 2a and 2b.

Figure 2a. The Image Viewer in viewing mode.

Figure 2b. The Image Viewer in viewing mode (fragment).

The Viewer is initially opened in viewing mode (Figures 2a and 2b). In this mode, the Viewer's toolbar includes dropbox of images is available for viewing, following by Layout and Edit toggles. When the Viewer page was loaded it requests Viewing Server for the list of images available for viewing. In our code sample, for the sake of simplicity it was assumed that Upload Server, images tile pyramid file storage and Viewing Server are located in the same machine. So to provide the client with the required list Viewer Server retrieves names of subfolders from a well-known folder containing all available image pyramids. Upon receiving it by the Web Viewer application, the list is placed to the client toolbar dropbox. The user selects an image name from the dropbox and views the image along with its annotation, if any. Image annotation is placed into a side panel which locates either to the left or to the right of the the image panel. The panel location may be switched with Layout toggle.

Working in viewing mode, the user can zoom and pan selected images and see already marked out selected regions along with their annotations. Our selected regions differ from native OSD overlays. According to OSD documentation [6] the overlays constitute HTML objects, wheres our selected regions are simply drawn on OSD viewer canvas. This approach seems to be more flexible permitting users to define non-rectangular selected regions and manage them dynamically as it will be shown below. When the user zooms and pans image, selected regions are moving accordingly sticking to the image similarly to OSD overlays. In our sample, selected region can be of three types: rectangular, circular and polygonal. When the user hovers the mouse pointer over the selected region in the image, the dynamic tooltip depicts the selected region title. For convenience of their management, selected regions are grouped to layers. All selected regions belong to the same layer may be shown / hidden either individually or by toggle the entire layer. Also, annotations of selected regions may be expanded / collapsed in similar manner.

By clicking Edit toggle user may switch Viewer from viewing mode to editing. In editing mode image is frozen in its current state (cannot be moved, zoomed or panned), and the user may edit the image. New instruments appear in both the toolbar (Figure 3) and the annotation table.

Figure 3. The Image Viewer toolbar in edit mode (with checkered flag idicating that new images are available).

Now it is possible to either edit an already existing selected region (changing its title, annotation and z-order within its layer) or create a new selected region and possibly a new layer. After editing completion user may save its result if she/he is satisfied with them. Alternatively user may undo editing to the point of the last save.

In order to create a new selected region, the user should select its type from Type dropbox (could be Rectangle, Circle or Polygon), choose the frame and background color and opacity with color picker control, define frame thickness (between 0 and 9 pixels) and than actually mark a selected region by mouse clicks on the image. In the case of rectangle and circle the selected region is defined by just two clicks (diagonal points for rectangle and center and a point defining radius for the circle). A polygon user may define as many points as she/he wants and at the and close polygon pressing appropriate button at the toolbox. After selected region shape has been created modal window to define its text information appears (depicted in Figure 4; the same window is used for editing of already existed selected region after appropriate toggle is clicked against required entry at annotation table). With this modal window a layer (either existed or newly created) may be assigned to the selected region.

Figure 4. Modal window to edit selected region.

The annotation table contains visual tools (Figure 5) to show / hide a selected region or the entire layer, change z-order of a selected region within its layer, edit text information (by opening edit modal window), or just delete selected region (layer also may be deleted if it does not contain any selected region).

Figure 5. Editing tools in Annotation table.

And now when we are familiar with functionality, let's briefly look at the code.

Code Description

The structure of the software is depicted in the flow chart at the beginning of the article (Figure 1). It consists of image upload and image viewing subsystems.

Upload

The upload subsystem consists of the Upload Server dzImageUpload, simple Web page ./dzImageUpload/index.html and a processor dzImagePyramidBuilder for building DZ image pyramid. dzImageUpload Web server is written in JS and running by node.js with express and multer packages. The server listens on port 15005 and supports HTTPS GET and POST requests. When user browses URL https://UploadServerURL:15005 Upload Server responds with a simple page ./dzImageUpload/index.html for the upload. With this page user selects image file and actually uploads it to the server. The upload is performed with HTTPS POST. The server's POST handler carries out the following tasks:

  • reload ./dzImageUpload/index.html page to browser for the next upload,
  • save uploaded file to disk using multer package, than
  • run dzImagePyramidBuilder to prepare tile pyramid for the uploaded image (this operation is currently available only in Windows OS - please see explanations below), and finally
  • notify Viewing Server dzImageView that a new image is ready for viewing and annotation.

For this matter dzImageUpload acts as a client of dzImageView using function

var req = https.request(options, function (res) {
               //.....
          });

with parameters

var agentOptions = {
    host: 'localhost',
    port: 15000,
    path: '/',
    rejectUnauthorized: false
};

var options = {
    url: 'https://localhost:15000',
    agent: new https.Agent(agentOptions),
    method: 'GET'
};

Because we assume that all components of our sample are running in the same machine, localhost IP addess may be used for inter-server communication.

The dzImagePyramidBuilder component is an image pyramid builder adopted from [3] and converted to .NET Core. In our sample, it is used as a black box and therefore its description is out of scope of this article. Explanations regarding this component structure may be found in [3] and algorithm used explaned in [1] and works on DZ imaging. The component was converted to .NET Core in order to make it compatible with different OS. But this became problematic. The issue is that dzImagePyramidBuilder uses System.Drawing assemply which is not currently supported by .NET Core. So the third party package CoreCompat.System.Drawing [7]. was chosen for replacement. It works fine in Windows but being deployed to Linux failed due to lack of Linux ready made support for GDI+ internally used by the library. So although the image upload works fine in Linux, pyramid preparation from the uploaded files is not available in our sample under Linux. Because the main focus of this article is DZ image viewing and annotation I allowed myself not to dig deeper for GDI+ support in Linux (your suggestions will be most welcome!) Also OSD recommends several Linux compatible image pyramid builders in its dedicated page [8]. I believe that Microsoft will provide support for its System.Drawing assembly and particularly for Bitmap type in upcoming releases of .NET Core.

Viewing, Mark Out and Annotation

Viewing subsystem is the main part of the code sample. It consists of dzImageView Web server written in JS for node.js with express package and ./dzImageView/dzweb Web single page application (SPA) written in JS and partly in TypeScript (TS).

dzImageView Web server listens on port 15000. When user browses URL https://ViewingServerURL:15000 the server sends back ./dzImageView/dzweb/index.html page starting dzweb Web application. The application may run with two parameters, namely,

  • lang=xx - language of UI, where "xx" stands for the two leading letters of the language, e.g. "en" for English (it is default), and
  • layout=x - denotes mutual placement of image and annotation panels. By default, the annotation panel is placed to the right from the image panel. And if layout parameter is set fore either "l" or "L" then annotation panel is placed to the left from the image panel.

So the URL

https://ViewingServerURL:15000?lang=ru&layout=l

causes the viewer with UI in Russian language with annotation at the left-hand side of the browser window.

dzweb is developed using mostly plain JS and CSS. Standard and the third party packages are used only for WebSocket communication (socket.io), selected region color picker (jcolor) and of course for DZ itself (openseadragon). TS is used for selected region types desribed in the followng files in the ./dzImageView/dzweb/ts folder: base type in selectedRegion.ts and derived types in selectedRegionCircle.ts, selectedRegionPolygon.ts and selectedRegionRectangle.ts. TS files should be transpiled to JS offline as a part of system preparation. This can be done with tsc.exe transpiler (has to be installed) by the following command from ./dzImageView/dzweb/ts folder:

"C:\Program Files (x86)\Microsoft SDKs\TypeScript\2.0"\tsc.exe

The command uses the tsconfig.json file. For the user's convenience the above command is placed into command file tsc.cmd. Its output according to parameter "compilerOptions"."outDir" will be put to the ./dzImageView/dzweb/js/selectedRegion folder.

The bulk of our Web application is located in the ./dzImageView/dzweb/js folder. We are already familiar with the selectedRegion subfolder containing results files of TS transpiling. The dzweb application is designed with strict separation between HTML elements and business logic. Files dealing with HTML elements are placed to separate page subfolder and named page*.js. Files with code free of HTML tags located in the ./dzImageView/dzweb/js folder. Page*() functions are called by non-page code directly. Calls in opposite direction are performed with appropriate callback provided to page* code.

Let's have a brief look at dzweb code. Code execution starts with funtion startUp() from file global.js. It is called from HTML page with

<script src="js/mainDz.js" onload="startUp()"></script>

Function startUp() activates theSingleton object (file singleton.js). The theSingleton object is used to hold variables used accross the application thus avoiding global variables. In startUp() theSingleton starts WebSocket communication with server and requests information about available images with 'init' message. Communication with server via WebSocket is implemented in function exchangeWithServer() using Promise for asynchronous operations. It takes as parameters message name, object to send to server and a callback function to process server's response. When Web application received list of available images from server it creates PageInfo object (file ./dzImageView/dzweb/js/page/page.js) with new method passing list of images and callback getImageInfo() for getting detailed data of an image as parameters to PageInit(). When details for the first image have been received from server dzweb completes its initialization and shows the first image along with its mark out and annotation, if available. Below main functionality is described by files.

global.js provides global functions for the entire application. It also includes constants

const hostIPAddress = '$0';
const __isRightLayout = $1;

which dollar-leading placeholders are replaced by server with actual values taken from client request. This is important particularly for hostIPAddress parameter to provide Web application with a proper IP address of the server.

mainDz.js contains most of OSD related code and WebSocket message handler.

layerCollection.js and layer.js contain LayerCollection and Layer objects respectively handling layers and selected regions collections.

transform.js is for Transform object responsible for web to image and vice versa coordinates transform.

tip.js contains Tip object for drawing tooltips when mouse moves over image's selected regions.

There are some features that are worth to mention.

Sticking Selected Region to Image

Moving selected regions along with image ("sticking") is achieved mainly with two pieces of code. The following handler is attached to OSD viewer object update-viewport event (file mainDz.js).

viewer.addHandler('update-viewport', function () {
    loopThroughAllSelectedRegions(function (selectedRegion) {
        selectedRegion.draw(viewer);
    });
});

Function selectedRegion.draw(viewer) is implemented by base type SelectedRegion. This function calls function public drawInternal(vr: any, ctx: any) implemented by derived selected region types. drawInternal() includes the following point coordinates transform:

Transform.prototype.imageToWeb = function (viewer, imagePoint) {
    var vpt = viewer.viewport;
    var ptWeb = vpt.pixelFromPoint(vpt.imageToViewportCoordinates(imagePoint), 
		    	    true); //!
			
    return osdPoint(ptWeb.x, ptWeb.y);
}

The second argument of function pixelFromPoint of viewer.viewport should be set to true indicating current point's location.

Iconic Font Awesome

This glyph font [9] is used to decorate visual tools, particularly buttons. In the sample only fontawesome-webfont.ttf font file is used. Font files with extensions woff and woff2 were removed from installation, and Viewing Server reports error on attempt to download them. These error messages may be ignored.

Localization

As it was stated above user may specify parameter lang=xx defining UI language. Viewing Server downloads file ./dzImageView/dzweb/settings/settings_xx.js as part of Viewing Web application. By default English language file is used. For simplicity the setting file currently contains all settings expected by the Web application (e.g. colors for tooltips). Currently the Russian language setting file is available for test along with the English one.

Mutually Exclusive Editing

When the user starts to edit a particular image all other users are not allowed to edit this image (toggle Edit disappears from toolbar for this image) until the user who "locked" the image will abandon edit mode for the image. If the user-editor saved editing results then server updates clients who currently view the image. It is possible that user-editor closes Web application being in edit mode. This can "lock" editing image until server restart. To avoid this problem and "unlock" the image permitting other users to edit it, the Viewing Server periodically (the period is difined by constant socketsCheckTimeoutInSec and currently set to 10 sec.) and on event of a new client connection, checks validity of all clients connections, removing invalid ones and unlocks images edited by unconnected clients. Please note that this is only the sample, and a real world system should have some permisson mechanism for editing and update.

New Image Availability Notification

When new image has been processed to a tile pyramid and become ready to viewing, the server sends notification to all running clients. The checkered flag sign at the right of toolbar indicates that new image is available and will be included to the image list on the next reload of Web page.

Running Sample

Node.js should be installed to run both upload and viewing servers.

Windows

In order to run upload subsystem .NET Core should be installed to run dzImagePyramidBuilder component. The sample was tested with .NET Core version 1.0.0. Viewing subsystem requires no prerequisites. Course of action is following.

  1. Unzip files to some root folder.
  2. Go to folder ./dzImagePyramidBuilder/src/dzImagePyramidBuilder and run file publish.cmd containing the following command:
    dotnet publish -o ../../../dzImageUpload/pyramidBuilder -c Release

    It will build dzImagePyramidBuilder and publish it to ./dzImageUpload/pyramidBuilder folder.

  3. Optional step. Go to folder ./dzNodeServer/dzweb/ts and run tsc.cmd command file containing command:
    "C:/Program Files (x86)/Microsoft SDKs/TypeScript/2.0"/tsc.exe

    to transpile TS files from this folder to ./dzNodeServer/dzweb/js/selectedRegion folder. Actually these JS files are already in place.

    Note: Please do not forget to transpile TS files if you will change them.

  4. Run Upload.cmd command file from the root folder. It starts dzImageUpload server with
    node server

    command from ./dzImageUpload folder.

    Note: The above command runs upload and viewing servers in both Windows and Linux environments.

  5. Open Web Browser (Google Chrome is recommended) and browse URL https://localhost:15005 and upload image file of your choice. The image will be placed to ./dzImageUpload/uploads folder replacing blanks with underscores and adding time in ticks to its name. Image tile pyramid will be output to appropriate subfolder under ./dzImageView/dzweb/images.
  6. Run View.cmd command file from the root folder. It starts dzImageView server from ./dzImageView folder.
  7. Browse URL https://localhost:15000 [optionally extended with: ?lang=en&layout=l] where en may be replaced with ru for the Russian language UI or file prepared by you in your favorite language. Viewer Web application screen depicted in Figures 2a and 2b will appear showing your uploaded image. For editing click on Edit toggle. After your editing is completed save its results by pressing Save button and leave edit mode clicking Edit toggle again.

     

    Debugging the servers and pyramid builder may be carried out opening ./dz.sln solution with Microsoft Visual Studio 2015.

Linux

As it was said above pyramid builder in its current version can not run in Linux. So despite image upload works fine there is no reason to run upload subsystem in Linux. Thus only view subsystem is discussed. Version Ubuntu 16.04 is used.

Here I describe the steps necessary to perform after installation in Windows described in previous section.

  1. Copy Windows root folder to appropriate root folder in Linux (only subfolder dzImageView should be actually copied to the root folder).
  2. Copy already prepared in Windows image pyramids to ./dzImageView/dzweb/images folder.
  3. Run dzImageView server.
  4. View / Edit images as it is described in point 7 for Windows installation.

Further Development

As I have already written above, this software cannot be viewed as a full blown product. So bugs of any sort are more than likely (e.g. circular selected region currently may occupy space outside the image). And of course, there is endless room for UI improvements. Selected region with smooth boundary (perhaps Bezier curve) may be added. Developer API will be useful particularly to assign some custom handlers to events related to selected regions, like click, hover, etc. Current version successfully works with Google Chrome and Mozilla Firefox. In Microsoft browsers (Internet Explorer and Edge) tooltips mechanism should be fixed. Layout may be adjusted to mobile devices.

Conclusion

This work presents software for large image smart viewing, mark out and annotation. Deep Zoom technology is used for viewing large images with high resolution without delay and latency. Upload and Viewing Servers were developed in Node.js with JavaScript being therefore platform independent. Deep Zoom technology implies usage of image tile pyramid. To prepare those pyramids separate component is used (its current version works only in Windows due to lack of GDI+ ready made support in Linux). Image Viewing Web application was developed with HTML/CSS, JavaScript and TypeScript with minimum third party packages.

References

[1] Deep Zoom. Wikipedia.
[2] Seadragon Software. Wikipedia.
[3] Igor Ladnik. Image in Azure: Upload, Deep Zoom Viewing, Mark Out and Annotation. CodeProject.
[4] OpenSeadragon. CodePlex.
[5] OpenSeadragon API.
[6] OpenSeadragon: Overlays.
[7] System.Drawing for .NET Core.
[8] OpenSeadragon: Creating Zooming Images.
[9] Iconic Font Awesome.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Igor Ladnik
Software Developer (Senior)
Israel Israel


  • Nov 2010: Code Project Contests - Windows Azure Apps - Winner

  • Feb 2011: Code Project Contests - Windows Azure Apps - Grand Prize Winner



You may also be interested in...

Comments and Discussions

 
Praisenice Pin
Member 1124779614-Jun-17 9:31
memberMember 1124779614-Jun-17 9:31 
GeneralRe: nice Pin
Igor Ladnik14-Jun-17 18:17
professionalIgor Ladnik14-Jun-17 18:17 
GeneralMy vote of 5 Pin
AJSON5-Jan-17 23:50
mvpAJSON5-Jan-17 23:50 
GeneralRe: My vote of 5 Pin
Igor Ladnik6-Jan-17 2:48
professionalIgor Ladnik6-Jan-17 2:48 
Questionvery nice Pin
Sacha Barber7-Dec-16 2:43
mvpSacha Barber7-Dec-16 2:43 
AnswerRe: very nice Pin
Igor Ladnik7-Dec-16 9:26
professionalIgor Ladnik7-Dec-16 9:26 
QuestionMy vote is 5 Pin
Pirks13-Dec-16 20:06
memberPirks13-Dec-16 20:06 
AnswerRe: My vote is 5 Pin
Igor Ladnik3-Dec-16 20:22
professionalIgor Ladnik3-Dec-16 20:22 
GeneralMy vote of 5 Pin
Pirks13-Dec-16 20:04
memberPirks13-Dec-16 20:04 
GeneralRe: My vote of 5 Pin
Igor Ladnik3-Dec-16 23:12
professionalIgor Ladnik3-Dec-16 23:12 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

Permalink | Advertise | Privacy | Terms of Use | Mobile
Web04 | 2.8.170915.1 | Last Updated 1 Dec 2016
Article Copyright 2016 by Igor Ladnik
Everything else Copyright © CodeProject, 1999-2017
Layout: fixed | fluid