Loading and Modifying Images, Part 1

by Tom Trinko

While Java can do many things, it's often used for making pretty pictures on the Web. In this two-part article, we'll learn how to build a picture object that loads and modifies images. We'll also learn how to use interfaces to provide support for callbacks, how to use the Runnable interface to produce classes that have Threads, how to use the MediaTracker class to load images, and how to convert the images to arrays of pixels, which we can manipulate using the PixelGrabber class. We'll see simple examples such as blur and unsharp mask.

Because we're covering so much ground in this article, we'll cover everything except the actual image manipulation . Next month, we'll see how to use PixelGrabber and those routines that can apply generalized convolutions, such as blur, to an image. This month's code will work just fine, but it will only load the image, not modify it.

Overview

First, we'll build a simple test Applet that loads and displays a GIF file. It will then modify the GIF file by blurring it, and display the modified image. In addition, we'll develop a reusable and extensible picture class, that will handle loading and modifying images. Let's discuss the classes we'll use, image_tools and picture.

image_tools

This is the test Applet. It sets things up, including creating an instance of the picture class, then lets the picture instance do most of the work.

picture

This class is a nice black box for getting and processing images. It's designed to be easily extensible so that we can quickly add new image processing functions. It hides all of the details of loading an image so all we have to do is pass a filename into it and get an image from it. A picture instance lets the Applet that created it know when an image is ready to be displayed by invoking a callback method that's supported by the Applet.

We'll use an interface, called picture_user. This interface contains the declarations of the callback methods so instances of picture can be used with any class that extends (i.e., inherits from) Applet and implements picture_user .

The basic Java toolset contains a number of classes that make it very easy to work with images. Two of the most valuable are MediaTracker, which handles the asynchronous loading of images, and PixelGrabber, which lets us convert an image into an array of pixels we can manipulate. As we'll see in the sample Applet, both of these classes are easy to use.

MediaTracker addresses the problem of simplifying the asynchronous loading of images. Essentially, we tell MediaTracker that we're loading an image and it tells us when the image is fully loaded. We'll use a GIF file in this example, but we could just as easily have used a JPEG formatted file.

PixelGrabber is designed to convert a Java image, or a rectangular piece of an image, into an array of pixels. Each pixel is an int containing the Alpha, Red, Green, and Blue values for the pixel. The first two bytes of the int are the Alpha channel, used for transparency and other effects; the second two are for red; the third two are for green; and the last two are for blue. This allows each channel or color to have a value ranging from 0 to 255.

Once you have an array of pixels, you can create methods that manipulate the image just as PhotoShop does. In this example, we'll apply a blurring function to an image and we'll also see the code needed to apply a variety of unsharp masks that enhance detail.

The sample Applet

Now let's walk through the sample Applet to see how all this stuff really works. We'll begin with the framework.

The Applet framework: example1.html

In order to run an Applet, we need an HTML file that will cause a browser or applet viewer to display the Applet. It doesn't need to be anything fancy, as we can see from the HTML shown in Listing A.

Listing A: The HTML to display the Applet

<HTML>
<HEAD>
  <META NAME="GENERATOR" CONTENT="Adobe PageMill 2.0 Mac">
  <TITLE>Java Image Processing Applet</TITLE>
</HEAD>
<BODY>

<H1><IMG SRC="mact.gif" WIDTH="133" HEIGHT="146" 
ALIGN="BOTTOM" NATURALSIZEFLAG="3"></H1>

<P><APPLET CODE="image_tools.class" CODEBASE="./" 
WIDTH="400" HEIGHT="300" ALIGN="BOTTOM"></APPLET>
</BODY>
</HTML>

.The first thing to note is that this HTML displays the image we're going to use in the Applet using the IMG tag. While this tag isn't necessary, it does insure that the GIF file is available in case we get an error in the Applet.

The actual Applet tag is very simple. The CODE parameter specifies the name of the file containing the class that inherits from Applet. The CODEBASE parameter specifies the location of the .class file relative to the location of the .html file. The WIDTH and HEIGHT parameters control the area reserved for our Applet, while the ALIGN parameter controls how the Applet is placed. This particular HTML assumes that both the GIF file and the class file are in the same directory as the .html file.

Defining callbacks: picture_user

Before we define our Applet class, we need to define an interface that contains the callback methods we want the picture class to invoke the Applet class. To see why we need to do this, imagine that we defined a class some_class that extends Applet. We then put a method in some_class, called a_callback.

Now we create a new class called picture. If we pass an instance of some_class to picture so picture can call the callback method, then our version of picture will only work with instances of class some_class. Take a look at what the definition of picture's creator method would look like:

public class picture(some_class the_instance)

Clearly, this setup wouldn't work with a different class, say some_other_class that extended Applet, because the input parameter's type would be different. This limitation isn't earth-shattering if we have access to the picture source code; however, it would be much nicer if we could design picture so that any subclass of Applet could work with it.

Fortunately, the designers of Java have developed a solution to this type of problem: We can use an interface. An interface is just a definition of a set of methods and attributes. While very similar to a class, an interface doesn't have the actual code for the methods, just their interface definitions—such as public void do_something(int x, double y, string the_name). If you're familiar with CORBA, just think of a Java interface as very similar to an IDL definition.

The nice thing about interfaces is that, while a class can only extend (be a child of) one other class, it can implement any number of interfaces. So in our example above, if the interface containing the callbacks that the picture class wanted to use was called picture_user, we could define our some_class like this:

public class some_class extends Applet implements picture_user {

and we could define the picture class creator method with the line

picture(picture_user my_parent) {

This way, any Applet class that implements the picture_user interface can use instances of the picture class without having to change the code of the picture class. The picture_user interface we'll use in this example is shown in Listing B.

Listing B: The callback interface

interface picture_user {
   public void picture_loaded(picture picture_getter);
   public void modified_picture_loaded(picture picture_getter);
   public void picture_error(picture picture_getter);
}

This interface is stored in a file called, not surprisingly, picture_user.java. The picture_loaded method is called by a picture instance when the base image is loaded, while modified_picture_loaded is called when a change to the base image, say a blur, is completed and ready to display.

The picture_error method is called whenever an error occurs in anything that the picture instance is doing: loading an image, for example. Since these methods are implemented in our class, we have complete control of what we do with them, even when we don't have the source code for the picture class.

The test framework: Image_tools class

This class inherits from Applet, and it implements the picture_user interface. It has just one thread of execution, so it can do only one thing at a time. We'll configure image_tools to first create an instance of the picture class, then have that instance load a base image; the instance will display that image, have the picture instance make a blurred copy of the base image, and finally, display the blurred image next to the original image.

The first thing we do is define what Java classes the Applet imports. We can then specify the class and its attributes, as shown in Listing C.

Listing C Setting up the image_tools class

import java.applet.Applet;
import java.awt.*;

public class image_tools extends Applet implements picture_user{
//the picture instance we use to get the //image and its modified version
   picture picture_getter;
//the original image
   Image the_picture;
//the modified image
   Image the_modified_picture;
//true if the modified image is available //for display
   boolean mod_picture_available_flag;
//true if the original picture is //available for display
   boolean picture_available_flag;

The main difference between extending a class and implementing an interface is that, while you get all of the methods of the extended class for free, you have to implement all of the methods specified in the inherited interface. For example, the Applet class has a getDocumentBase method, which returns the location of the HTML page in which Applet is contained. When your class extends Applet, you can use that method by just invoking it.

On the other hand, methods in an interface aren't inherited. So the picture_loaded method of the picture_user interface can't be called until we write it. Interfaces are useful for callbacks, as we'll see in this example, but their primary use is to effectively provide a similar functionality to multiple inheritance—where a class inherits from more than one parent class.

The image_tool attributes contain the picture instance that we'll use for loading and manipulating images, as well as the images that are used, and some flags for indicating when images are available. The flags are used to let the Applet know when it can draw the images to the screen.

Initializing the Applet

The init method, shown in Listing D, initializes the Applet and creates a new picture instance. It then tells the instance to start loading a picture using the load_picture method.

Listing D: Setting the starting values for the Applet

/** this initializes the Applet */
  public void init(){
//at the beginning the pictures aren't //available
     picture_available_flag = false;
     mod_picture_available_flag = false;
//create the picture instance and specify //which image(GIF or JPEG) file to use
     picture_getter = new       picture("mact.gif",this);
//start the fetching of the base image
     picture_getter.load_picture();
  }

Is the basic image loaded?: picture_loaded

There are several ways the Applet could find out from the picture class when the picture is loaded. We could create a new thread in the Applet and periodically poll the picture instance to see if the picture is done. The approach we'll use is to put a callback routine in the Applet, which is called by the picture instance when the image is finished loading. As we saw before, this method is defined in the picture_user interface.

Listing E shows the picture_loaded implementation, which is called by the picture instance when the image is fully loaded. While the signature of the callback method is fixed, you can put whatever code you want inside the method.

For our example, we first get the loaded image that was so we can display it, using the Applets paint method describedin the next section. Next, we set a flag so the other methods in the Applet know the base image is loaded. Then, we tell the Applet to repaint itself so the new image will be displayed as soon as it's loaded.

Finally, we start the next step in the image processing, which is to create a blurred version of the base image. We do that by telling the picture instance to blur the base image that it has loaded. (Next month, we'll describe how to do the blur and the meaning of the parameters, but for now the key thing is that the blur operation is asynchronous—i.e., the call to blur will return before the blurred image is ready.) If the blur method returns false, then an error has occurred.

Listing E: Loading the original image

//this call back routine is called //by picture when the base picture is //loaded
   public void picture_loaded(picture   
                      a_picture_getter) {
//get the image that was loaded
      the_picture = a_picture_getter.get_picture();
      System.out.println("image is " + the_picture);
//let the rest of the Applet know the //image was loaded
      picture_available_flag = true;
//make sure the image is displayed as //soon as possible
      repaint();
//blur the base image
       if( !picture_getter.blur(1,true)) {
   System.out.println("problem loading blurred picture");
       }
   }

Is the modified image loaded?: modified_picture_loaded

The last thing the picture_loaded method did was start the processing required to generate a blurred copy of the base image. The modified_picture_loaded method, shown in Listing F, is a callback that's invoked by the picture instance when the modified image is ready to be displayed.

Listing F: Loading the modified picture (see Part II for more info)

//this call back routine is called when a //modified picture is loaded   
   public void modified_picture_loaded(picture a_picture_getter) {
//get the modified image
   the_modified_picture = a_picture_getter.get_modified_picture();
//check for problems
   if (the_modified_picture == null) {
//look for errors
   System.out.println("Error:image_tools:Modified image not loaded");
   } else {
//let the rest of the Applet that the //modified image is available
   mod_picture_available_flag = true;
//display the modified image as soon as //possible
    repaint();
       }
}

The method gets the newly loaded image, making sure it's not null. Then it has the Applet display it in the paint method.

Was there a problem?: picture_error

This callback is used when an error occurs loading an image. Listing G shows a very simple implementation—you'd need a more sophisticated one for any real program.

Listing G: A callback for dealing with errors

public void picture_error(picture picture_getter) {
//here we can do our error handling
   System.out.println("error loading picture");
}

Redrawing the screen: paint

The paint method, which is called when we invoke the repaint method, draws the Applets interface. Listing H shows our paint method, which checks to see if the images are available. If an image is available, it's displayed; if the image isn't available, we show the user a message indicating that he should be patient.

Keeping the user informed is very important on the Web because the wide variety of network connections, client computers, and network congestion that users frequently deal with can leave them wondering what's going on. No matter how wonderful our images, if the user gets frustrated waiting for them to appear and leaves the site, we've failed.

Listing H: Drawing what the user sees

public void paint(Graphics g){
//if the base picture is available //display it, if not display a message to //let the user know what's going on
   if(picture_available_flag) {
   g.drawImage(the_picture,0,0,this);
    } else {
   g.drawString("Please Wait for 
                       Picture",50,50);
    }
//if the modified image is available //display it, if not let the user know //what's going on
   if(mod_picture_available_flag) {
   g.drawImage(the_modified_picture,
                           150,150,this);
    } else {
   g.drawString("Please Wait for 
              Modified Picture",130,130);
    }
}

Working with images: picture class

The real work in this example is done in the picture class. A picture class instance provides a one-stop shopping center for image acquisition and modification. You'll be able to easily extend this class to provide more complex, unique, or better-looking image processing by just adding new methods.

A picture instance has five possible states:

no image loaded
base image loading
just the base image loaded
base image loaded, modified image loading
base and modified image loaded

The only constraint on the order in which these states occur is that a base image must be loaded before anything can be done with a modified image. Listing I shows the imports, class definition, and attributes of the picture class.

Listing I: Setting up the picture class

import java.awt.image.*;
import java.awt.*;
import java.applet.Applet;

/** This class is designed to load and process images. Each instance can work with one base picture and one
processed one. */
public class picture implements Runnable{
//the thread used by the instance
   Thread my_thread;
//MediaTracker handles the asynchronous //loading of images
   MediaTracker my_tracker;
//this array contains the pixel data from //the base image
   int pixels[];
//this array is used as a temporary //working area for modifications
//to the image
   int working_pixels[];
//this array contains the pixels for the //modified image
    int modified_pixels[];
//these are the dimensions of the base //image
   int x_max, y_max;
//this is the base image
   Image the_image;
//this is the modified image
   Image the_modified_image;
//true if we're loading the base image
   boolean loading_image_flag;
//this is the name of the file that //contains the base image
   String the_file_name;
//this is the instance that created the //picture instance, used for 
//invoking call backs
   picture_user my_parent;
//this is true when we're converting an //image to pixels
   boolean pixels_loading_image_flag;
//true when we're converting the modified //pixels to an image
   boolean loading_modified_image_flag;
//true if there's a valid base image //loaded
   boolean image_loaded_flag;
//true if there is a valid modified image //loaded
   boolean mod_image_loaded_flag;
//true if the pixel array contains a //valid data
   boolean pixels_loaded_flag;
//true if the modified pixel array is //loaded
   boolean mod_pixels_loaded_flag;
//contains the weights used in a //convolution
   double pixel_weights[];

There are a number of interesting things in Listing I. First, the picture class implements the Runnable interface, which allows us to have a thread without having to extend the Thread class. While we could have just inherited from the Thread class, implementing Runnable allows us to have picture inherit from some other class in the future. Feel free to reimplement this by extending Thread if you'd like to experiment.

Next we define attributes for a MediaTracker instance. The MediaTracker instance will handle all the tough work involved in loading an image, converting a GIF or JPEG file into something Java can display, for example.

The three arrays of ints will contain the pixels of the base image, modified image, and a working array used during image transformations. We'll set the size of the arrays when we read in the base image.

One last thing to note in Listing I is that we keep a reference to the image_tool class that created us. This reference allows us to invoke the callback routines. You might be wondering why we've used all of those boolean flags that define the state of the instance (images being loaded, etc.) rather than a single value with a set of constants, such as picture.IMAGE_LOADING.

This decision is purely a style issue. You'll find it's easier if you can tell from the attribute name and its value (i.e., true or false) what's going on rather than having to compare an attribute to a set of constants. Feel free to rewrite this class using a single state variable if you find that variable is easier to read.

Creating a new Instance: picture

The picture class is designed to be associated with an image file, GIF or JPEG, and a parent class instance that supports the required callbacks. Initializing an instance involves setting the starting values of the various flags and creating a MediaTracker instance, as shown in Listing J.

Listing J: Creating a new picture instance

/**This method creates a new instance of the picture class 
@param file_name  This is the name of the file to be loaded as the base image
@param name parent  The instance that is to be called when the processing is done */
picture(String file_name,picture_user parent) {
   the_file_name = file_name;
   loading_image_flag = false;
   pixels_loading_image_flag = false;
   loading_modified_image_flag = false;
   pixels_loaded_flag = false;
   mod_pixels_loaded_flag = false;
   my_parent = parent;
//create a new media tracker and give it 
//an Applet to serve as the required 
//component.
   my_tracker = new MediaTracker((Applet)my_parent);
//start the thread for this instance //running
   start();
}

The last thing the creator method does is start the instances thread, through the start method.

Running in Parallel: start

This method creates and starts a thread that will contain the instances' activities, as shown in Listing K.

Listing K: Starting to watch for when the images are ready

/** start the thread */
public void start() {
  if (my_thread == null) {
    my_thread = new Thread(this);
     my_thread.start();
  }
}

We want to make sure that we don't create a new thread if one already exists. The final call starts the new thread.

Getting an image over the net: load_picture

This is the first method our Applet should call. The only information it needs, as we can see from Listing L, is the name of the file. The file name is defined in Listing D, provided as the name of the file—assuming it's in the same directory as the Applet class files. We have to set the pixels_loaded_flag to false, because we might decide to load more than one base image, each replacing the previous one. In that case, the pixels_loaded_flag will be true when load_picture is called.

Listing L: Starting the image-loading activity

/** load the base picture */
public void load_picture() {
 //mark that we're loading the base image
  loading_image_flag = true;
//mark that pixels aren't loaded for the //base image
  pixels_loaded_flag = false;
//mark that the image we requested is not //yet loaded
  image_loaded_flag = false;
//get the image which is in the same //folder as the Applet
 the_image = ((Applet)my_parent).getImage(((Applet)my_parent).getDocumentBase(),the_file_name);
//start tracking the loading of the image
  my_tracker.addImage(the_image,0);
}

We use the getImage method, which is defined on the Applet class, to fetch the image. Because we bring in the parent Applet as an instance of picture_user, we need to cast my_parent to Applet in order to use Applet methods. This also means that the picture class can only work with Applets.

We could use picture with an Application by checking whether my_parent was an instance of Applet, and using a different set of methods, such as the Toolkit classes getImage, to load the image. Inside the getImage parameter list, we use another Applet method, getDocument, to get the URL of the HTML page that contains the Applet.

The getImage method returns before the image is fully loaded. This situation is good, because it means your Applet doesn't have to sit idle while waiting for the picture to load. But it also means we've got to handle this asynchronous activity and figure out when the image is completely loaded.

Fortunately, Sun gave us the MediaTracker class. To use MediaTracker, just call the addImage method with the image and an integer ID, as in the last line in Listing L. If you want to track multiple images together, just assign them the same ID number. The rest of the work is done in the run method, where we periodically poll the MediaTracker instance to see if the image is fully loaded.

Polling for Pictures: run

The run method, shown in Listing M, wakes up periodically and looks to see whether an image has finished loading. We use the MediaTracker checkID method to determine that the image is loaded. The first parameter to checkID is the number we used when we called the addImage method. The second parameter for checkID tells MediaTracker to load the image if it's not already being loaded. If the image is loaded, then we invoke the appropriate callback on the Applet.

We can vary the time between checks by changing the parameter in the sleep method. In this case we check every 500 milliseconds (i.e., every 1/2 second). The more frequently we poll, the more CPU cycles we use, but we get less delay between the time the image is loaded and the time we find out.

Listing M: Watching for image-loading to complete

public void run() {
//run forever.  If the time interval //between waking up is long this is no //burden on the computer.  
  while (true) {
    try {
      my_thread.sleep(500); 
    } catch(InterruptedException e) {
   System.out.println("ERROR:picture:run: problem with sleeping " + e);
    }
//The first time through after we've //started to load the base picture that //it's finished loading we get the image //size, initialize the pixel array, and //call the picture_loaded callback in the //instance that created this picture //instance
    if(my_tracker.checkID(0,true) & loading_image_flag) {
     if (!my_tracker.isErrorID(0)) {
//once the image is loaded we can get its //dimensions
   x_max = the_image.getWidth((Applet)my_parent);
   y_max = the_image.getHeight((Applet)my_parent);
//initialize the pixel array in case we do //image processing with the image
   pixels = new int[x_max * y_max];
//initialize the temporary pixel array
   init_working_image();
//we're done loading the image and we //don't want to execute this section
//of the code until we load another base //image
   loading_image_flag = false;
//mark that the base image is loaded
   image_loaded_flag = true;
//this lets the parent react to the //picture being loaded      my_parent.picture_loaded(this);

     } else {
//handle the case that an error occurred
   loading_image_flag = false;
   image_loaded_flag = false;
   my_parent.picture_error(this);
     }
   }
//the first time through the while loop //after the modified image is loaded we //call the parent instance so it can use //the new loaded image
   if(my_tracker.checkID(1,true) & loading_modified_image_flag) {
     loading_modified_image_flag = false;
      mod_image_loaded_flag = true;
      my_parent. modified_picture_loaded(this);
      }
    }//end while
}

Notice how we use the isErrorID method to see if an error occurred while the MediaTracker class is loading the image. If an error occurs, we'll call the picture_error callback routine and set the appropriate flags to mark the new state of the instance.

The one tricky thing we have to do here is initialize the pixel arrays after the base image loads. We have to wait until the image is completely loaded, because we don't know how big it is until that point. If we try to get the image size before the picture is fully loaded, we'll get incorrect results.

There's another way to use MediaTracker when you load images. We could have used the waitForID or waitForAll methods to wait for the images to be loaded. Neither of these methods return until the applet loads the image or images in question. We could create a thread whose only purpose in life is to wait for waitForID to return.

And one other critical thing you should know about MediaTracker is that every MediaTracker instance retains references to every image it loads. As a result, Java can't garbage collect image instances—which can be very large—as long as the MediaTracker instance that loaded it is around. If you plan to use picture to load a series of base images, you should delete the old MediaTracker instance and create a new one for every picture—the load_picture method is a natural place to do this.

Getting a loaded image: get_picture

Once the image is loaded, the parent Applet uses this method to get the image. The method, shown in Listing N, makes sure that the image is loaded. If it's not—because it's in the process of being loaded, an error occurred, or the Applet hasn't asked the picture instance to load an image—null is returned. A better piece of code would return an int, whose allowable values would be defined in the picture class as a set of constants that identified the problem type.

Listing N: Giving the image to the Applet

/** This accessor method returns the base image. If the image is being loaded then null is returned. */
   public Image get_picture() {
//if the image is being loaded or no //image is loaded return null
     if(loading_image_flag && !image_loaded_flag) {
       return null;
     } else {
       return the_image;
     }
      }

Notes

Although the examples we explored in this article were fairly simple, more complex ones are possible. For example, check out my neon glow Applet at

http://members.aol.com/trinkos/javaintro.html.

Please note that if you want to run this month's code, you need to add the dummy methods in Listing O.

Listing O: Modified image method stubs

public boolean blur(int x, boolean y) {
   return false;
}
public Image get_modified_picture() {
   return null;
}
public void init_working_image() {
}

You'll still get an error message from the program, since we don't actually produce a modified image, but you'll see the base image load. We'll complete and explain these methods in next month's issue.

Conclusion

We've seen how to build interfaces to support the use of callbacks which allow coordination of asynchronous processes—an important factor in the multi-processor world of today. We've also seen how to use MediaTracker to manage the asynchronous loading of images, which is important when you're working with variable load times over the Internet or your intranet. Next month we'll finish off the picture object by seeing how we can take the base image, loaded by MediaTracker, and apply various image processing techniques to it and then display the modified version.

Copyright © 1998, ZD Inc. All rights reserved. ZD Journals and the ZD Journals logo are trademarks of ZD Inc. Reproduction in whole or in part in any form or medium without express written permission of ZD Inc. is prohibited. All other product names and logos are trademarks or registered trademarks of their respective owners.