2010-11-30

Java Continuations and Green Threads at native speeds

Continuations, or green threads, are a concept mostly seen in scripting languages. Green threads are just like native threads, but you can run multiple of them on a single native thread. Java has no native support for continuations, so we have to transform (instrument) java bytecode with a library that saves localvars and the instruction pointer upon a GreenThread.yield(). Once the green thread is resumed, this information is used to rebuild the stacktrace, with all localvars intact.

The library used is: http://l33tlabs.org/Continuations/ written by Matthias Mann
His library depends on ASM3 to parse and construct the bytecode: http://asm.ow2.org/

Through the Java Instrumentation Agent that I wrote for the previously mentioned library, there is no need to transform your byteclass ahead of time. The java agent will take care of this at runtime.

java -javaagent:conti4.jar -cp ./your.jars your.MainClass


I wrote a wrapper around the continuation library, to lower the bar a bit. I hope the code pretty much speaks for itself, and if not, feel free to ask questions. For additional convenience, I added some demo code and its fascinating output at the bottom.

Enjoy!
import de.matthiasmann.continuations.Coroutine;
import de.matthiasmann.continuations.CoroutineProto;
import de.matthiasmann.continuations.SuspendExecution;
import de.matthiasmann.continuations.Coroutine.State;

/*
 * Created on Nov 30, 2010
 * 
 * @author Riven
 */

public abstract class GreenThread implements CoroutineProto
{
   public static final long EOF = -1L;

   private final Coroutine  coroutine;

   public GreenThread()
   {
      this.coroutine = new Coroutine(this);
   }

   private long doSleep;

   @Override
   // this is where you put your code
   public abstract void coExecute() throws SuspendExecution;

   long step()
   {
      coroutine.run();
      if (coroutine.getState() == State.FINISHED)
         return EOF;
      return this.doSleep;
   }

   public static void yield() throws SuspendExecution
   {
      GreenThread.sleep(0L);
   }

   public static void sleep(long ms) throws SuspendExecution
   {
      GreenThread thread = (GreenThread) Coroutine.getActiveCoroutine().getProto();

      thread.doSleep = Math.max(ms, 0L);

      Coroutine.yield();
   }
}



---------------------------------------------------------------------

import java.util.Comparator;
import java.util.PriorityQueue;

/*
 * Created on Nov 30, 2010
 * 
 * @author Riven
 */

public class GreenThreadQueue
{
   private final PriorityQueue queue;
   private final List          reschedule;

   public GreenThreadQueue()
   {
      this.queue = new PriorityQueue(16, new GreenThreadWakeupComparator());
      this.reschedule = new ArrayList();
   }

   public void start(GreenThread thread)
   {
      this.queue.add(new GreenThreadWakeup(thread, 0L));
   }

   public boolean tick(long now)
   {
      try
      {
         while (true)
         {
            GreenThreadWakeup wakeup = this.queue.peek();
            if (wakeup == null)
               return !this.reschedule.isEmpty(); // signal nothing more to do

            if (wakeup.timestamp > now)
               break;

            if (this.queue.poll() != wakeup)
               throw new IllegalStateException();

            long sleep = wakeup.thread.step();
            if (sleep == GreenThread.EOF)
               continue;

            wakeup.timestamp = now + sleep;
            this.reschedule.add(wakeup);
         }

         return true;
      }
      finally
      {
         for (GreenThreadWakeup wakeup : this.reschedule)
         {
            this.queue.add(wakeup);
         }
         this.reschedule.clear();
      }
   }

   static private class GreenThreadWakeup
   {
      public final GreenThread thread;
      public long              timestamp;

      public GreenThreadWakeup(GreenThread thread, long timestamp)
      {
         this.thread = thread;
         this.timestamp = timestamp;
      }
   }

   static class GreenThreadWakeupComparator implements Comparator
   {
      @Override
      public int compare(GreenThreadWakeup o1, GreenThreadWakeup o2)
      {
         int val = Long.signum(o1.timestamp - o2.timestamp);
         return (val == 0) ? 1 : val;
      }
   }
}



Demo code
GreenThreadQueue queue = new GreenThreadQueue();

      GreenThread thread1 = new GreenThread()
      {
         @Override
         public void coExecute() throws SuspendExecution
         {
            for (int i = 0; i < 3; i++)
            {
               System.out.println("a" + i);
               GreenThread.sleep(1500);
            }
         }
      };

      GreenThread thread2 = new GreenThread()
      {
         @Override
         public void coExecute() throws SuspendExecution
         {
            for (int i = 0; i < 4; i++)
            {
               System.out.println("b" + i);
               GreenThread.sleep(1300);
            }
         }
      };

      queue.start(thread1);
      queue.start(thread2);

      do
      {
         try { Thread.sleep(100); } catch(InterruptedException exc) { /* ignore */ }
      }
      while (queue.tick(System.currentTimeMillis()));
   }
Fancy output (from 1 thread)
a0
b0
b1
a1
b2
a2
b3

2010-11-24

Lost four words... (pro nouns)

It got cold while she called for juice from , not awaiting the tide and tied a tight knot. A scene never seen again, maybe missed due to the dew or mist on site by a dude with poor sight on one side. A phase etched on the edge of his face. Whether her patients had patience depended on the weather. So take a look at the sea and see: sow and pour it in a poor pore, with any luck, lock it. I'll weigh the way the rained isle was reigned. A pane hid me, then a pain hit me. During the war the witch wore a hat which was won by one bright bride. During the juring descent, she thought and fought the scent. Worn by the sun, I warn my son, as his eye tans tense, 'caus where regular wear hides hides, I knew a new gnu (its dead dad had that dept) that played ball, sipping a bowl for four hours, assuming it was ours, while the cued cheap cute sheep queued. Silence. Deaf by impending death. 'Hear here', she heard hurt. She waited, weighted by lead, led down the stairs by the wrath filled failed ref. Bye Susan, by all means, awe at the whirled world... Eight dodo's ate dough though.

cold/called
juice/
not/knot
tide/tight/tied
scene/seen
missed/mist
due/dew
site/side/sight
phase/face
etch/edge

whether/weather
patience/patients
suck it/socket
rained/reighed
thought/fought
eight/ate
dodo/though/dough
so/sow
luck/lock
sea/see
pour/pore/poor

war/wore
won/one
witch/which
bright/bride
worn/warn
sun/son
tans/tense
where/wear
knew/new/gnu
dead/dad/that/dept
descent/the scent

ball/bowl
for/four
hours/ours
cued/cute/queued
cheap/sheep
plug/plaque
deaf/death
hear/here
heard/hurt
waited/weighted

lead/led
wrath/ref
filled/failed
bye/by
all/awe
whirled/world
i'll/isle
way/weigh
pain/pane
hit/hid
Disclaimer: English is not my native language, so certain words might sound the same for me, while they are clearly distinct for you gays.

2010-08-19

Calculate PerlinNoise faster with an optimization to grad()

A simple optimization to the gradient function makes the noise function twice as fast.

PerlinNoise.grad()
inline float grad(int hash, float x, float y, float z)
   {  
 //float u = (h < 8) ? x : y;
 //float v = (h < 4) ? y : ((h == 12 || h == 14) ? x : z);
 //return ((h & 1) == 0 ? u : -u) + ((h & 2) == 0 ? v : -v);

 switch(hash & 0xF)
 {
  case 0x0: return  x + y;
  case 0x1: return -x + y;
  case 0x2: return  x - y;
  case 0x3: return -x - y;
  case 0x4: return  x + z;
  case 0x5: return -x + z;
  case 0x6: return  x - z;
  case 0x7: return -x - z;
  case 0x8: return  y + z;
  case 0x9: return -y + z;
  case 0xA: return  y - z;
  case 0xB: return -y - z;
  case 0xC: return  y + x;
  case 0xD: return -y + z;
  case 0xE: return  y - x;
  case 0xF: return -y - z;
  default: return 0; // never happens
 }
   }

PerlinNoise.noise() (click code to expand)
int   perm[512];

   void initPerlinNoise()
   {
      int i, permutation[] = { 151, 160, 137, 91, 90, 15, 131, 13, 201, 95, 96, 53, 194, 233, 7, 225, 140, 36, 103, 30, 69, 142, 8, 99, 37, 240, 21, 10, 23, 190, 6, 148, 247, 120, 234, 75, 0, 26, 197, 62, 94, 252, 219, 203, 117, 35, 11, 32, 57, 177, 33, 88, 237, 149, 56, 87, 174, 20, 125, 136, 171, 168, 68, 175, 74, 165, 71, 134, 139, 48, 27, 166, 77, 146, 158, 231, 83, 111, 229, 122, 60, 211, 133, 230, 220, 105, 92, 41, 55, 46, 245, 40, 244, 102, 143, 54, 65, 25, 63, 161, 1, 216, 80, 73, 209, 76, 132, 187, 208, 89, 18, 169, 200, 196, 135, 130, 116, 188, 159, 86, 164, 100, 109, 198, 173, 186, 3, 64, 52, 217, 226, 250, 124, 123, 5, 202, 38, 147, 118, 126, 255, 82, 85, 212, 207, 206, 59, 227, 47, 16, 58, 17, 182, 189, 28, 42, 223, 183, 170, 213, 119, 248, 152, 2, 44, 154, 163, 70, 221, 153, 101, 155, 167, 43, 172, 9, 129, 22, 39, 253, 19, 98, 108, 110, 79, 113, 224, 232, 178, 185, 112, 104, 218, 246, 97, 228, 251, 34, 242, 193, 238, 210, 144, 12, 191, 179, 162, 241, 81, 51, 145, 235, 249,
         14, 239, 107, 49, 192, 214, 31, 181, 199, 106, 157, 184, 84, 204, 176, 115, 121, 50, 45, 127, 4, 150, 254, 138, 236, 205, 93, 222, 114, 67, 29, 24, 72, 243, 141, 128, 195, 78, 66, 215, 61, 156, 180 };

      for (i = 0; i < 256; i++)
         perm[256 + i] = perm[i] = permutation[i];
   }


   inline float fade(float t)
   {
      return t * t * t * (t * (t * 6.0f - 15.0f) + 10.0f);
   }

   inline float lerp(float t, float a, float b)
   {
      return a + t * (b - a);
   }

   inline float grad(int hash, float x, float y, float z)
   {  
 //float u = (h < 8) ? x : y;
 //float v = (h < 4) ? y : ((h == 12 || h == 14) ? x : z);
 //return ((h & 1) == 0 ? u : -u) + ((h & 2) == 0 ? v : -v);

 switch(hash & 0xF)
 {
  case 0x0: return  x + y;
  case 0x1: return -x + y;
  case 0x2: return  x - y;
  case 0x3: return -x - y;
  case 0x4: return  x + x;
  case 0x5: return -x + x;
  case 0x6: return  x - x;
  case 0x7: return -x - x;
  case 0x8: return  y + x;
  case 0x9: return -y + x;
  case 0xA: return  y - x;
  case 0xB: return -y - x;
  case 0xC: return  y + z;
  case 0xD: return -y + x;
  case 0xE: return  y - x;
  case 0xF: return -y - z;
  default: return 0; // never happens
 }
   }

   static inline float __fastcall noise(float x, float y, float z)
   {
      jint ix,iy,iz,gx,gy,gz;
      jint a0,b0,aa,ab,ba,bb;

      float aa0,ab0,ba0,bb0;
      float aa1,ab1,ba1,bb1;
      float a1,a2,a3,a4,a5,a6,a7,a8;
      float u,v,w,a8_5,a4_1;

      ix = (jint)x; x-=ix;
      iy = (jint)y; y-=iy;
      iz = (jint)z; z-=iz;

      gx = ix & 0xFF;
      gy = iy & 0xFF;
      gz = iz & 0xFF;
      
      a0 = gy+perm[gx];
      b0 = gy+perm[gx + 1];
      aa = gz+perm[a0];
      ab = gz+perm[a0 + 1];
      ba = gz+perm[b0];
      bb = gz+perm[b0 + 1];

      aa0 = perm[aa]; aa1 = perm[aa + 1];
      ab0 = perm[ab]; ab1 = perm[ab + 1];
      ba0 = perm[ba]; ba1 = perm[ba + 1];
      bb0 = perm[bb]; bb1 = perm[bb + 1];

      a1 = grad(bb1, x-1, y-1, z-1);
      a2 = grad(ab1, x  , y-1, z-1);
      a3 = grad(ba1, x-1, y  , z-1);
      a4 = grad(aa1, x  , y  , z-1);
      a5 = grad(bb0, x-1, y-1, z  );
      a6 = grad(ab0, x  , y-1, z  );
      a7 = grad(ba0, x-1, y  , z  );
      a8 = grad(aa0, x  , y  , z  );

      u = fade(x);
      v = fade(y);
      w = fade(z);

      a8_5 = lerp(v, lerp(u, a8, a7), lerp(u, a6, a5));
      a4_1 = lerp(v, lerp(u, a4, a3), lerp(u, a2, a1));
      return lerp(w, a8_5, a4_1);
   }

2010-07-23

Rhino ClassShutter replacement: SandboxShutter

Because the ClassShutter in Rhino doesn't usually provide enough information to make decisions on whether or not certain Java objects, fields or methods should be accessible, I wrote the SandboxShutter which will enable you to control accessibility for each field and each method of each Java object. The performance impact is negligible, because JavaScript itself is an order of magnitude slower than the injected checks.


The SandboxShutter interface:
public interface SandboxShutter
{
   public boolean allowClassAccess(Class<?> type);

   public boolean allowFieldAccess(Class<?> type, Object instance, String fieldName);

   public boolean allowMethodAccess(Class<?> type, Object instance, String methodName);

   public boolean allowStaticFieldAccess(Class<?> type, String fieldName);

   public boolean allowStaticMethodAccess(Class<?> type, String methodName);
}


When the following javascript code is executed:
importPackage(Packages.my.game); // assuming the Java class my.game.Player exists

var player = new Player("Jake");
player.gender = "female"; // this is a Java method!
player.setGender("male"); // this the same Java method.
player.age = 18; // this is a Java field
player.age += 3;

var player = new Player("Jane");
player.gender = "male";
player.gender = "female";
player.age = 19;
player.age += 2;

var count = Player.PLAYER_COUNT;
Player.PLAYER_COUNT += 2;


The following SandboxShutter calls will be made:
allowClassAccess:       my.game.Player

allowMethodAccess:      my.game.Player.setGender() instance=Player@2346
allowFieldAccess:       my.game.Player.age         instance=Player@2346

allowMethodAccess:      my.game.Player.setGender() instance=Player@54326
allowFieldAccess:       my.game.Player.age         instance=Player@54326

allowStaticFieldAccess: my.game.Player.PLAYER_COUNT
As shown, there will be at most one call for each field/method in each object, and one call per class. This allows you to control accessibility per Java object.


Create the SandboxContextFactory:
   public static class SandboxContextFactory extends ContextFactory
   {
      final SandboxShutter shutter;

      public SandboxContextFactory(SandboxShutter shutter)
      {
         this.shutter = shutter;
      }

      @Override
      protected Context makeContext()
      {
         Context cx = super.makeContext();
         cx.setWrapFactory(new SandboxWrapFactory());
         cx.setClassShutter(new ClassShutter()
         {
            private final Map<String, Boolean> nameToAccepted = new HashMap<String, Boolean>();

            @Override
            public boolean visibleToScripts(String name)
            {
               Boolean granted = this.nameToAccepted.get(name);

               if (granted != null)
               {
                  return granted.booleanValue();
               }

               Class< ? > staticType;
               try
               {
                  staticType = Class.forName(name);
               }
               catch (Exception exc)
               {
                  this.nameToAccepted.put(name, Boolean.FALSE);
                  return false;
               }

               boolean grant = shutter.allowClassAccess(staticType);
               this.nameToAccepted.put(name, Boolean.valueOf(grant));
               return grant;
            }
         });
         return cx;
      }

      class SandboxWrapFactory extends WrapFactory
      {
         @Override
         public Scriptable wrapNewObject(Context cx, Scriptable scope, Object obj)
         {
            this.ensureReplacedClass(scope, obj, null);

            return super.wrapNewObject(cx, scope, obj);
         }

         @Override
         public Object wrap(Context cx, Scriptable scope, Object obj, Class< ? > staticType)
         {
            this.ensureReplacedClass(scope, obj, staticType);

            return super.wrap(cx, scope, obj, staticType);
         }

         @Override
         public Scriptable wrapAsJavaObject(Context cx, Scriptable scope, Object javaObject, Class< ? > staticType)
         {
            final Class< ? > type = this.ensureReplacedClass(scope, javaObject, staticType);

            return new NativeJavaObject(scope, javaObject, staticType)
            {
               private final Map<String, Boolean> instanceMethodToAllowed = new HashMap<String, Boolean>();

               @Override
               public Object get(String name, Scriptable scope)
               {
                  Object wrapped = super.get(name, scope);

                  if (wrapped instanceof BaseFunction)
                  {
                     String id = type.getName() + "." + name;
                     Boolean allowed = this.instanceMethodToAllowed.get(id);

                     if (allowed == null)
                     {
                        boolean allow = shutter.allowMethodAccess(type, javaObject, name);
                        this.instanceMethodToAllowed.put(id, allowed = Boolean.valueOf(allow));
                     }

                     if (!allowed.booleanValue())
                     {
                        return NOT_FOUND;
                     }
                  }
                  else
                  {
                     // NativeJavaObject + only boxed primitive types?
                     if (!shutter.allowFieldAccess(type, javaObject, name))
                     {
                        return NOT_FOUND;
                     }
                  }

                  return wrapped;
               }
            };
         }

         //

         private final Set<Class< ? >> replacedClasses = new HashSet<Class< ? >>();

         private Class< ? > ensureReplacedClass(Scriptable scope, Object obj, Class< ? > staticType)
         {
            final Class< ? > type = (staticType == null && obj != null) ? obj.getClass() : staticType;

            if (!type.isPrimitive() && !type.getName().startsWith("java.") && this.replacedClasses.add(type))
            {
               this.replaceJavaNativeClass(type, scope);
            }

            return type;
         }

         private void replaceJavaNativeClass(final Class< ? > type, Scriptable scope)
         {
            Object clazz = Context.jsToJava(ScriptableObject.getProperty(scope, "Packages"), Object.class);
            Object holder = null;
            for (String part : Text.split(type.getName(), '.'))
            {
               holder = clazz;
               clazz = ScriptableObject.getProperty((Scriptable) clazz, part);
            }
            NativeJavaClass nativeClass = (NativeJavaClass) clazz;

            nativeClass = new NativeJavaClass(scope, type)
            {
               @Override
               public Object get(String name, Scriptable start)
               {
                  Object wrapped = super.get(name, start);

                  if (wrapped instanceof BaseFunction)
                  {
                     if (!shutter.allowStaticMethodAccess(type, name))
                     {
                        return NOT_FOUND;
                     }
                  }
                  else
                  {
                     // NativeJavaObject + only boxed primitive types?
                     if (!shutter.allowStaticFieldAccess(type, name))
                     {
                        return NOT_FOUND;
                     }
                  }

                  return wrapped;
               }
            };

            ScriptableObject.putProperty((Scriptable) holder, type.getSimpleName(), nativeClass);
            ScriptableObject.putProperty(scope, type.getSimpleName(), nativeClass);
         }
      }
   }


Install the (global) SandboxContextFactory:
      ContextFactory.initGlobal(new SandboxContextFactory(new SandboxShutter()
      {
         ...
      }));

      // create and initialize Rhino Context
      Context cx = Context.enter();
      Scriptable prototype = cx.initStandardObjects();
      Scriptable topLevel = new ImporterTopLevel(cx);
      prototype.setParentScope(topLevel);
      Scriptable scope = cx.newObject(prototype);
      scope.setPrototype(prototype);

      // your scripts

2010-07-20

Functions on Iterables :: Consume

public class Functional
{
   /**
    * Consumes (removes) all items while iterating
    */

   public static <T> Iterable<T> consume(final Iterable<T> iterable)
   {
      if (iterable == null)
         throw new NullPointerException();

      return new Iterable<T>()
      {
         @Override
         public Iterator<T> iterator()
         {
            final Iterator<T> iterator = iterable.iterator();

            return new Iterator<T>()
            {
               @Override
               public boolean hasNext()
               {
                  return iterator.hasNext();
               }

               @Override
               public T next()
               {
                  T result = iterator.next();
                  iterator.remove();
                  return result;
               }

               @Override
               public void remove()
               {
                  throw new NoSuchElementException("already removed");
               }
            };
         }
      };
   }




To remove all short texts in a list:
List<String> data = ...;

Filter<String> shortText = new Filter<String>()
{
   public boolean accept(String item) { return item == null || item.length() <= 3; }
}

for(String item: consume(filter(data, shortText)))
{
   System.out.println("By the time you see this, '"+item+"' has been removed.");
}

Functions on Iterables :: Event Hooks

public interface Operator
{
   public void operate(T item);
}

public class Functional
{
   /**
    * Performs a callback on each element-visit, and each element removal
    */

    public static <T> Iterable<T> eventHook(final Iterable<T> iterable,
                                            final Operator<T> onVisit,
                                            final Operator<T> onRemove)
   {
      if (iterable == null)
         throw new NullPointerException();
      if (onVisit == null && onRemove == null)
         throw new NullPointerException("must specify either onVisit, onRemove or both");

      return new Iterable<T>()
      {
         @Override
         public Iterator<T> iterator()
         {
            final Iterator<T> iterator = iterable.iterator();

            return new Iterator<T>()
            {
               @Override
               public boolean hasNext()
               {
                  return iterator.hasNext();
               }

               @Override
               public T next()
               {
                  this.current = iterator.next();
                  if (onVisit != null)
                     onVisit.operate(this.current);
                  return this.current;
               }

               @Override
               public void remove()
               {
                  iterator.remove();
                  if (onRemove != null)
                     onRemove.operate(this.current);
               }

               //

               private T current;
            };
         }
      };
   }
}

Example:
Operator<String> onVisit= new Operator<String>()
{
   public void operate(String text)
   {
      System.out.println("visited: "+text);
   }
};

Operator<String> onRemove= new Operator<String>()
{
   public void operate(String text)
   {
      System.out.println("removed: "+text);
   }
};

Iterable<String> data = ...;

for(String item: eventHook(data, onVisit, onRemove))
{
   if(item.length() > String.valueOf(Math.PI))
      System.out.println("this text is longer than PI!");
}


Once you created an Iterable (say, a filtered view on a List) you can iterate over it as often as you want. No need to create a new Iterable.


With the support for closures in Java 7 a lot of the boilerplate code will disappear.

Functions on Iterables :: Transformer

public interface Transformer<I, O>
{
   public O transform(I item);
}

public class Functional
{
   /**
    * Transforms Iterable<I> into Iterable<O> using a Transformer<I, O>
    */

   public static <I, O> Iterable<O> transform(final Iterable<I> iterable,
                                              final Transformer<I, O> transformer)
   {
      if (iterable == null)
         throw new NullPointerException();
      if (transformer == null)
         throw new NullPointerException();

      return new Iterable<O>()
      {
         @Override
         public Iterator<O> iterator()
         {
            final Iterator<I> iterator = iterable.iterator();

            return new Iterator<O>()
            {
               @Override
               public boolean hasNext()
               {
                  return iterator.hasNext();
               }

               @Override
               public O next()
               {
                  return transformer.transform(iterator.next());
               }

               @Override
               public void remove()
               {
                  iterator.remove();
               }
            };
         }
      };
   }
}

Example:
Transformer<String, Integer> textToNumber = new Transformer<String, Integer>()
{
   public Integer transform(String text)
   {
      return Integer.valueOf(text);
   }
};

Iterable<String> data = ...;

for(Integer number: transform(data, textToNumber))
{
   System.out.println("number: "+number.intValue());
}

Functions on Iterables :: Filter

Functional programming has its strengths when iterating over data. If you want a view on a subset data, you can make a filter, and iterate over your data only seeing the elements that the filter accepted:

public interface Filter<T>
{
   public boolean accept(T value);
}

public class Functional
{
   /**
    * Filter items from the view of the returned Iterable<T>
    */

   public static <T> Iterable<T> filter(final Iterable<T> iterable, final Filter<T> filter)
   {
      if (iterable == null)
         throw new NullPointerException();
      if (filter == null)
         throw new NullPointerException();

      return new Iterable<T>()
      {
         @Override
         public Iterator<T> iterator()
         {
            final Iterator<T> iterator = iterable.iterator();

            return new Iterator<T>()
            {
               @Override
               public boolean hasNext()
               {
                  this.ensureReady();
                  return this.ready;
               }

               @Override
               public T next()
               {
                  this.ensureReady();
                  if (!this.ready)
                     throw new NoSuchElementException();

                  T result = this.current;
                  this.ready = false;
                  this.current = null;
                  return result;
               }

               @Override
               public void remove()
               {
                  iterator.remove();
               }

               //

               private boolean ready = false;
               private T       current;

               private void ensureReady()
               {
                  while (!this.ready && iterator.hasNext())
                  {
                     T item = iterator.next();
                     if (!filter.accept(item))
                        continue;

                     this.ready = true;
                     this.current = item;
                  }
               }
            };
         }
      };
   }
}

Example:
Filter<String> nonNull = new Filter<String>()
{
   public boolean accept(String value)
   {
      return value != null;
   }
};

Iterable<String> data = ...;

for(String item: filter(data, nonNull))
{
   System.out.println("item: "+item); // guaranteed not to be null
}

2010-04-29

Histogram :: array based

For any reasonably sized Histogram, you probably want to look at this Histogram class based on a HashMap. However, if your histograms are tiny, say, less than 8 elements occur frequently, this array based histogram is an order of magnitude faster.

import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.NoSuchElementException;

public class TinyHistogram<T>
{
   private T[]   items;
   private int[] usage;
   private int   size;

   public TinyHistogram()
   {
      this.items = (T[]) new Object[4];
      this.usage = new int[4];
      this.size = 0;
   }

   public int put(T item)
   {
      if (item == null)
         throw new IllegalArgumentException();

      int io = this.indexOfItem(item);

      if (io != -1)
      {
         int result = ++this.usage[io];
         this.swapIncreasedValue(io);
         return result;
      }

      // add item to the end
      if (this.size == this.items.length)
         this.grow();
      this.items[this.size] = item;
      this.usage[this.size] = 1;
      this.size += 1;
      return 1;
   }

   public int get(T item)
   {
      if (item == null)
         throw new IllegalArgumentException();

      int io = this.indexOfItem(item);
      if (io == -1)
         return 0; // not -1, as the object occurred zero times
      return this.usage[io];
   }

   public List<T> topKeys()
   {
      return this.topKeys(Integer.MAX_VALUE);
   }

   public List<T> topKeys(int amount)
   {
      int end = Math.min(this.size, amount);

      // make a copy, the items are already sorted
      List<T> tops = new ArrayList<T>();
      for (int i = 0; i < end; i++)
         tops.add(this.items[i]);
      return tops;
   }

   public List<Pair<T, Integer>> topEntries()
   {
      return this.topEntries(Integer.MAX_VALUE);
   }

   public List<Pair<T, Integer>> topEntries(int amount)
   {
      int end = Math.min(this.size, amount);

      // make a copy, the items are already sorted
      List<Pair<T, Integer>> tops = new ArrayList<Pair<T, Integer>>();
      for (int i = 0; i < end; i++)
         tops.add(new Pair<T, Integer>(this.items[i], Integer.valueOf(this.usage[i])));
      return tops;
   }

   public int remove(T item)
   {
      int io = this.indexOfItem(item);
      if (io == -1)
         throw new NoSuchElementException(String.valueOf(item));

      int result = --this.usage[io];
      if (result == 0)
         return this.removeIndex(io);

      this.swapDecreasedValue(io);
      return result;
   }

   public int reset(T item)
   {
      int io = this.indexOfItem(item);
      if (io == -1)
         throw new NoSuchElementException(String.valueOf(item));
      return this.removeIndex(io);
   }

   //

   private final void swapDecreasedValue(int index)
   {
      while ((index + 1) < size && usage[index + 0] < usage[index + 1])
      {
         int a = index + 0;
         int b = index + 1;

         swapObj(items, a, b);
         swapInt(usage, a, b);

         index++;
      }
   }

   private final void swapIncreasedValue(int index)
   {
      while (index > 0 && usage[index + 0] > usage[index - 1])
      {
         int a = index - 0;
         int b = index - 1;

         swapObj(items, a, b);
         swapInt(usage, a, b);

         index--;
      }
   }

   private static final void swapInt(int[] array, int a, int b)
   {
      int temp = array[a];
      array[a] = array[b];
      array[b] = temp;
   }

   private static final void swapObj(Object[] array, int a, int b)
   {
      Object temp = array[a];
      array[a] = array[b];
      array[b] = temp;
   }

   private void grow()
   {
      this.items = Arrays.copyOf(this.items, this.items.length * 2);
      this.usage = Arrays.copyOf(this.usage, this.usage.length * 2);
   }

   private int removeIndex(int index)
   {
      int count = this.usage[index];
      int shift = --this.size - index;
      System.arraycopy(this.items, index + 1, this.items, index, shift);
      System.arraycopy(this.usage, index + 1, this.usage, index, shift);
      return count;
   }

   private int indexOfItem(T item)
   {
      for (int i = 0; i < this.size; i++)
         if (this.items[i] == item || this.items[i].equals(item))
            return i;
      return -1;
   }
}

Slicing direct buffers to workaround 4K overhead

Direct ByteBuffers can have an unexpected massive overhead. For every allocation, a number of bytes equal to the system pagesize (typically 4K) is added. The reason for this is that MappedByteBuffers must be page-aligned. The code used behind the scenes in ByteBuffer.allocateDirect(int size) looks a little something like this:

private static ByteBuffer malloc(int size)
{
   int pageSize = unsafe.getPageSize(); // typically 4096
   long pointer = sun.misc.Unsafe.malloc(size + pageSize);
   long base = (pointer + (pageSize-1)) / pageSize * pageSize;
   ByteBuffer buffer = unsafe.createBufferAt(base);
   return buffer;
}

Code like this will have a massive overhead:
for(int i=0; i<count; i++)
   buffers[i] = ByteBuffer.allocateDirect(64);

Instead use an approach like below:
public class DirectBufferProvider
{
   private static final int ALLOCATION_SIZE = 1024*1024;

   private ByteBuffer currentBuffer = null;

   public ByteBuffer allocate(int size)
   {
      if(size >= ALLOCATION_SIZE)
         return ByteBuffer.allocateDirect(size);

      if(currentBuffer == null || size > currentBuffer.remaining())
         currentBuffer = ByteBuffer.allocateDirect(ALLOCATION_SIZE);

      currentBuffer.limit(currentBuffer.position() + size);
      ByteBuffer result = currentBuffer.slice();
      currentBuffer.position(currentBuffer.limit());
      currentBuffer.limit(currentBuffer.capacity());
      return result;
   }



   private static DirectBufferProvider global_synced = new DirectBufferProvider();

   public static ByteBuffer allocateDirect(int size)
   {
      synchronized(global_synced)
      {
         return global_synced.allocate(size);
      }
   }
}

Note that unlike ByteBuffer.allocateDirect(), the code in DirectBufferProvider.allocate() is not threadsafe. The static (convenience) method is synchronized and should thus not be used in heavily multithreaded code. Using ThreadLocals is even worse, performance wise, so just make new DirectBufferProvider instances when you need them.

2010-02-23

Delaying security dialogs in Java until you need them

Delaying security dialogs in Java until you need them, is a bit harder than it should be. By default, the dialog appears before the first line of code is executed, scaring off your casual visitor.

If you only occasionally need to perform actions that require elevated privileges, you can delay the security dialog to the absolute last moment (for example: right before reading/writing a file). The trick is to keep most of your code in an unsigned JAR, and the code that requires elevated privileges into a signed JAR. Use Class.forName(String) to load the signed class, which will prompt the security dialog.

Using an interface in the unsigned code, and the implementation in the signed code, you can keep your code tidy.


Note:

The browser will remember the choice of the user, until a *restart* of the browser. To workaround this (when people declined), create a dozen tiny signed JARs (with a dozen different certificates, mind you) and use a roundrobin algorithm, using serverside code or javascript that generates the applet-archive attribute. After a dozen hits and rejections, you can be sure your visitor will never grant access to his system anyway.

Update:

To make it work in MSIE, both classes MUST be in separate packages.


IMPORTANT: Since 6u19 this doesn't work anymore. You not only get a rather confusing security dialog (clicking [YES] means deny access, clicking [NO] means allow access), the two classes end up in different classloaders that cannot access eachother, resulting in ClassNotFoundException / NoClassDefFoundException. Thanks Oracle, for making Java's user experience even more secure and crap at the same time.
http://java.sun.com/javase/6/docs/technotes/guides/jweb/mixed_code.html



On to the code, which is reasonably simple.

Unsigned JAR:
package some.unsigned.stuff;

public interface SecureAccess
{
   public byte[] loadFile(File file);
   public void storeFile(File file, byte[] data);
}

     // Usage:

     File file = new File("/home/silly/image.jpg");
     Class< ? > clazz = Class.forName("some.signed.stuff.LocalSecureAccess");
     SecureAccess access = (SecureAccess) clazz.newInstance();
     byte[] data = access.loadFile(file);

Signed JAR:
package some.signed.stuff;

public class LocalSecureAccess implements SecureAccess
{
   public byte[] loadFile(final File file)
   {
      return AccessController.doPrivileged(new PrivilegedAction<byte[]>()
      {
         @Override
         public byte[] run()
         {
            return loadFileImpl(file);
         }
      });
   }

   @Override
   public void storeFile(final File file, final byte[] data)
   {
      AccessController.doPrivileged(new PrivilegedAction<Object>()
      {
         @Override
         public Object run()
         {
            storeFileImpl(file, data);

            return null;
         }
      });
   }

   // implementation

   static final int MAX_FILE_SIZE = 8 * 1024 * 1024; // prevent applet running out of memory

   byte[] loadFileImpl(File file)
   {
      DataInputStream input = null;

      try
      {
         long len = file.length();
         if (len > MAX_FILE_SIZE)
            throw new IllegalStateException("file too big: " + file);

         byte[] data = new byte[(int) len];
         input = new DataInputStream(new FileInputStream(file));
         input.readFully(data);
         input.close();
         return data;
      }
      catch (IOException exc)
      {
         throw new IllegalStateException(exc);
      }
      finally
      {
         try { if(input!=null) input.close(); } catch(IOException exc) {}
      }
   }

   void storeFileImpl(File file, byte[] data)
   {
      OutputStream output = null;

      try
      {
         output = new FileOutputStream(file);
         output.write(data);
         output.flush();
      }
      catch (IOException exc)
      {
         throw new IllegalStateException(exc);
      }
      finally
      {
         try { if(output!=null) output.close(); } catch(IOException exc) {}
      }
   }
}

Jar signing 101: (using DSA keys instead of RSA for Java 1.4 compatibility)
PATH=%PATH%;path\to\JDK\bin
SET ALIAS=MY_ALIAS
SET PASS=MY_PASSWORD
SET JAR=my.jar

keytool -delete -storepass %PASS% -alias %ALIAS%
keytool -genkey -storepass %PASS% -keypass %PASS% -keyalg DSA -alias %ALIAS%
   -dname "CN=full.domainname.com, OU=Your unit, O=Your Company,
           L=Your city, ST=Your state, C=CA,
           EMAILADDRESS=your@server.com DC=server, DC=com"
   -validity 999 (put all of this on one line)
keytool -selfcert -storepass %PASS% -alias %ALIAS% -validity 999
keytool -exportcert -storepass %PASS% -alias %ALIAS% -rfc -file %ALIAS%.cer
jarsigner -storepass %PASS% -keypass %PASS% %JAR% %ALIAS%
pause

Applet code: (nothing special)
    <applet
      code="package/of/YourApplet.class"
      archive="unsigned.jar,signed.jar"
      width="640"
      height="480">
      no applet?
    </applet>  

2010-02-04

Image :: Java Animated GIFs (with transparant pixel disposal modes)

It's a nightmare to find the code to create animated GIFs in Java. Once you found it, you notice you can't set the frame disposal, and transparent pixels will show pixels in the previous frames. The following code snippet solves this.

public class GifFrame
{
   public static final String NONE                = "none";
   public static final String DO_NOT_DISPOSE      = "doNotDispose";
   public static final String RESTORE_TO_BGCOLOR  = "restoreToBackgroundColor";
   public static final String RESTORE_TO_PREVIOUS = "restoreToPrevious";

   public final BufferedImage img;
   public final long          delay; // in millis
   public final String        disposalMethod;

   public GifFrame(BufferedImage img, long delay)
   {
      this(img, delay, NONE);
   }

   public GifFrame(BufferedImage img, long delay, String disposalMethod)
   {
      this.img = img;
      this.delay = delay;
      this.disposalMethod = disposalMethod;
   }
}

public class ImageUtil
{
   public static BufferedImage convertRGBAToGIF(BufferedImage src, int transColor)
   {
      BufferedImage dst = new BufferedImage(src.getWidth(), src.getHeight(), BufferedImage.TYPE_BYTE_INDEXED);
      Graphics g = dst.getGraphics();
      g.setColor(new Color(transColor));
      g.fillRect(0, 0, dst.getWidth(), dst.getHeight());
      {
         IndexColorModel indexedModel = (IndexColorModel) dst.getColorModel();
         WritableRaster raster = dst.getRaster();
         int sample = raster.getSample(0, 0, 0);
         int size = indexedModel.getMapSize();
         byte[] rr = new byte[size];
         byte[] gg = new byte[size];
         byte[] bb = new byte[size];
         indexedModel.getReds(rr);
         indexedModel.getGreens(gg);
         indexedModel.getBlues(bb);
         IndexColorModel newModel = new IndexColorModel(8, size, rr, gg, bb, sample);
         dst = new BufferedImage(newModel, raster, dst.isAlphaPremultiplied(), null);
      }
      dst.createGraphics().drawImage(src, 0, 0, null);
      return dst;
   }

   public static void saveAnimatedGIF(OutputStream out, List<GifFrame> frames, int loopCount) throws Exception
   {
      ImageWriter iw = ImageIO.getImageWritersByFormatName("gif").next();

      ImageOutputStream ios = ImageIO.createImageOutputStream(out);
      iw.setOutput(ios);
      iw.prepareWriteSequence(null);

      int p = 0;
      for (GifFrame frame : frames)
      {
         ImageWriteParam iwp = iw.getDefaultWriteParam();
         IIOMetadata metadata = iw.getDefaultImageMetadata(new ImageTypeSpecifier(frame.img), iwp);
         ImageUtil.configureGIFFrame(metadata, String.valueOf(frame.delay / 10L), p++, frame.disposalMethod, loopCount);
         IIOImage ii = new IIOImage(frame.img, null, metadata);
         iw.writeToSequence(ii, null);
      }

      iw.endWriteSequence();
      ios.close();
   }

   private static void configureGIFFrame(IIOMetadata meta, String delayTime, int imageIndex, String disposalMethod, int loopCount)
   {
      String metaFormat = meta.getNativeMetadataFormatName();

      if (!"javax_imageio_gif_image_1.0".equals(metaFormat))
      {
         throw new IllegalArgumentException("Unfamiliar gif metadata format: " + metaFormat);
      }

      Node root = meta.getAsTree(metaFormat);

      Node child = root.getFirstChild();
      while (child != null)
      {
         if ("GraphicControlExtension".equals(child.getNodeName()))
            break;
         child = child.getNextSibling();
      }

      IIOMetadataNode gce = (IIOMetadataNode) child;
      gce.setAttribute("userDelay", "FALSE");
      gce.setAttribute("delayTime", delayTime);
      gce.setAttribute("disposalMethod", disposalMethod);

      if (imageIndex == 0)
      {
         IIOMetadataNode aes = new IIOMetadataNode("ApplicationExtensions");
         IIOMetadataNode ae = new IIOMetadataNode("ApplicationExtension");
         ae.setAttribute("applicationID", "NETSCAPE");
         ae.setAttribute("authenticationCode", "2.0");
         byte[] uo = new byte[] { 0x1, (byte) (loopCount & 0xFF), (byte) ((loopCount >> 8) & 0xFF) };
         ae.setUserObject(uo);
         aes.appendChild(ae);
         root.appendChild(aes);
      }

      try
      {
         meta.setFromTree(metaFormat, root);
      }
      catch (IIOInvalidTreeException e)
      {
         throw new Error(e);
      }
   }
}

Usage:
List<GifFrame> gifFrames = new ArrayList<GifFrame>();

   for(BufferedImage image: images)
   {
      int transparantColor = 0xFF00FF; // purple
      BufferedImage gif = convertRGBAToGIF(image, transparantColor);
      long delay = 100; // every frame takes 100ms
      String disposal = GifFrame.RESTORE_TO_BGCOLOR; // make transparent pixels not 'shine through'
      gifFrames.add(new GifFrame(gif, delay, disposal));
   }

   int loopCount = 0; // loop indefinitely
   saveAnimatedGIF(outputStream, gifFrames, loopCount);

Image :: read/write TGA

public static BufferedImage readTGA(File file) throws IOException
   {
      if (!file.exists())
         throw new FileNotFoundException(file.getAbsolutePath());

      byte[] header = new byte[18];
      int len = (int) file.length() - header.length;
      if (len < 0)
         throw new IllegalStateException("file not big enough to contain header: " + file.getAbsolutePath());
      byte[] data = new byte[len];

      RandomAccessFile raf = new RandomAccessFile(file, "r");
      raf.read(header);
      raf.read(data);
      raf.close();

      if ((header[0] | header[1]) != 0)
         throw new IllegalStateException(file.getAbsolutePath());
      if (header[2] != 2)
         throw new IllegalStateException(file.getAbsolutePath());
      int w = 0, h = 0;
      w |= (header[12] & 0xFF) << 0;
      w |= (header[13] & 0xFF) << 8;
      h |= (header[14] & 0xFF) << 0;
      h |= (header[15] & 0xFF) << 8;

      boolean alpha;
      if ((w * h) * 3 == data.length)
         alpha = false;
      else if ((w * h) * 4 == data.length)
         alpha = true;
      else
         throw new IllegalStateException(file.getAbsolutePath());
      if (!alpha && (header[16] != 24))
         throw new IllegalStateException(file.getAbsolutePath());
      if (alpha && (header[16] != 32))
         throw new IllegalStateException(file.getAbsolutePath());
      if ((header[17] & 15) != (alpha ? 8 : 0))
         throw new IllegalStateException(file.getAbsolutePath());

      BufferedImage dst = new BufferedImage(w, h, alpha ? BufferedImage.TYPE_INT_ARGB : BufferedImage.TYPE_INT_RGB);
      int[] pixels = ((DataBufferInt) dst.getRaster().getDataBuffer()).getData();
      if (pixels.length != w * h)
         throw new IllegalStateException(file.getAbsolutePath());
      if (data.length != pixels.length * (alpha ? 4 : 3))
         throw new IllegalStateException(file.getAbsolutePath());

      if (alpha)
      {
         for (int i = 0, p = (pixels.length - 1) * 4; i < pixels.length; i++, p -= 4)
         {
            pixels[i] |= ((data[p + 0]) & 0xFF) << 0;
            pixels[i] |= ((data[p + 1]) & 0xFF) << 8;
            pixels[i] |= ((data[p + 2]) & 0xFF) << 16;
            pixels[i] |= ((data[p + 3]) & 0xFF) << 24;
         }
      }
      else
      {
         for (int i = 0, p = (pixels.length - 1) * 3; i < pixels.length; i++, p -= 3)
         {
            pixels[i] |= ((data[p + 0]) & 0xFF) << 0;
            pixels[i] |= ((data[p + 1]) & 0xFF) << 8;
            pixels[i] |= ((data[p + 2]) & 0xFF) << 16;
         }
      }

      if ((header[17] >> 4) == 1)
      {
         // ok
      }
      else if ((header[17] >> 4) == 0)
      {
         // flip horizontally

         for (int y = 0; y < h; y++)
         {
            int w2 = w / 2;
            for (int x = 0; x < w2; x++)
            {
               int a = (y * w) + x;
               int b = (y * w) + (w - 1 - x);
               int t = pixels[a];
               pixels[a] = pixels[b];
               pixels[b] = t;
            }
         }
      }
      else
      {
         throw new UnsupportedOperationException(file.getAbsolutePath());
      }

      return dst;
   }

   public static void writeTGA(BufferedImage src, File file) throws IOException
   {
      DataBuffer buffer = src.getRaster().getDataBuffer();
      boolean alpha = src.getColorModel().hasAlpha();
      byte[] data;

      if (buffer instanceof DataBufferByte)
      {
         byte[] pixels = ((DataBufferByte) src.getRaster().getDataBuffer()).getData();
         if (pixels.length != src.getWidth() * src.getHeight() * (alpha ? 4 : 3))
            throw new IllegalStateException();

         data = new byte[pixels.length];

         for (int i = 0, p = pixels.length - 1; i < data.length; i++, p--)
         {
            data[i] = pixels[p];
         }
      }
      else if (buffer instanceof DataBufferInt)
      {
         int[] pixels = ((DataBufferInt) src.getRaster().getDataBuffer()).getData();
         if (pixels.length != src.getWidth() * src.getHeight())
            throw new IllegalStateException();

         data = new byte[pixels.length * (alpha ? 4 : 3)];

         if (alpha)
         {
            for (int i = 0, p = pixels.length - 1; i < data.length; i += 4, p--)
            {
               data[i + 0] = (byte) ((pixels[p] >> 0) & 0xFF);
               data[i + 1] = (byte) ((pixels[p] >> 8) & 0xFF);
               data[i + 2] = (byte) ((pixels[p] >> 16) & 0xFF);
               data[i + 3] = (byte) ((pixels[p] >> 24) & 0xFF);
            }
         }
         else
         {
            for (int i = 0, p = pixels.length - 1; i < data.length; i += 3, p--)
            {
               data[i + 0] = (byte) ((pixels[p] >> 0) & 0xFF);
               data[i + 1] = (byte) ((pixels[p] >> 8) & 0xFF);
               data[i + 2] = (byte) ((pixels[p] >> 16) & 0xFF);
            }
         }
      }
      else
      {
         throw new UnsupportedOperationException();
      }

      byte[] header = new byte[18];
      header[2] = 2; // uncompressed, true-color image
      header[12] = (byte) ((src.getWidth() >> 0) & 0xFF);
      header[13] = (byte) ((src.getWidth() >> 8) & 0xFF);
      header[14] = (byte) ((src.getHeight() >> 0) & 0xFF);
      header[15] = (byte) ((src.getHeight() >> 8) & 0xFF);
      header[16] = (byte) (alpha ? 32 : 24); // bits per pixel
      header[17] = (byte) ((alpha ? 8 : 0) | (1 << 4));

      RandomAccessFile raf = new RandomAccessFile(file, "rw");
      raf.write(header);
      raf.write(data);
      raf.setLength(raf.getFilePointer()); // trim
      raf.close();
   }

FastMath :: fast floor + ceil

Calling Math.floor() simply takes too long. The problem with optimizing is that you can't simply cast the floatingpoint number to an integer, because that will result in invalid results for negative numbers. If we know our input values are in a specific range, we can safely add a certain (big) constant to the input, cast the guaranteed positive value to an integer and subtract the constant again.

The code is ~9x faster than Math.floor(). Replacing the doubles with floats makes it faster, but the results are rather... random, so don't.

public class FastMath
{
   private static final int    BIG_ENOUGH_INT   = 16 * 1024;
   private static final double BIG_ENOUGH_FLOOR = BIG_ENOUGH_INT + 0.0000;
   private static final double BIG_ENOUGH_ROUND = BIG_ENOUGH_INT + 0.5000;
   private static final double BIG_ENOUGH_CEIL  = BIG_ENOUGH_INT + 0.9999;

   public static int fastFloor(float x) {
      return (int) (x + BIG_ENOUGH_FLOOR) - BIG_ENOUGH_INT;
   }

   public static int fastRound(float x) {
      return (int) (x + BIG_ENOUGH_ROUND) - BIG_ENOUGH_INT;
   }

   public static int fastCeil(float x) {
      return (int) (x + BIG_ENOUGH_CEIL) - BIG_ENOUGH_INT;
   }
}