make[1]: Entering directory `/usr/src/linux-headers-3.2.0-2-amd64'
  CC [M]  /tmp/vmware-root/modules/vmnet-only/userif.o
  CC [M]  /tmp/vmware-root/modules/vmnet-only/netif.o
  CC [M]  /tmp/vmware-root/modules/vmnet-only/filter.o
/tmp/vmware-root/modules/vmnet-only/userif.c: In function ‘VNetCsumCopyDatagram’:
/tmp/vmware-root/modules/vmnet-only/userif.c:520:3: error: incompatible type for argument 1 of ‘kmap’
/usr/src/linux-headers-3.2.0-2-common/include/linux/highmem.h:48:21: note: expected ‘struct page *’ but argument is of type ‘const struct ’
/tmp/vmware-root/modules/vmnet-only/userif.c:523:3: error: incompatible type for argument 1 of ‘kunmap’
/usr/src/linux-headers-3.2.0-2-common/include/linux/highmem.h:54:20: note: expected ‘struct page *’ but argument is of type ‘const struct ’
/tmp/vmware-root/modules/vmnet-only/netif.c: In function ‘VNetNetIfSetup’:
/tmp/vmware-root/modules/vmnet-only/netif.c:134:7: error: unknown field ‘ndo_set_multicast_list’ specified in initializer
/tmp/vmware-root/modules/vmnet-only/netif.c:134:7: warning: initialization from incompatible pointer type [enabled by default]
/tmp/vmware-root/modules/vmnet-only/netif.c:134:7: warning: (near initialization for ‘vnetNetifOps.ndo_validate_addr’) [enabled by default]
make[4]: *** [/tmp/vmware-root/modules/vmnet-only/userif.o] Error 1
make[4]: *** Waiting for unfinished jobs....
make[4]: *** [/tmp/vmware-root/modules/vmnet-only/netif.o] Error 1

After lots of Googling I found a blog post with a patch for kernels 3.2.0 and 3.3.0. Unfortunately when I tried to run the patch it failed and said:

Sorry, this script is only for VMWare WorkStation 8.0.2 or VMWare Player 4.0.2. Exiting

In order to fix this open up the script after you download it and change the line this line:

vmreqver=8.0.2

To this:

vmreqver=8.0.3

Re-run the script and you should be good to go.

With some creative Googling I found two invaluable resources that I need to give credit to:

• Monitoring Declarative Transactions in Spring
• Starting new transactions in Spring bean

Here’s what I distilled out of everything I went through:

1. @Transactional annotations only work on public methods. If you have a private or protected method with this annotation there’s no (easy) way for Spring AOP to see the annotation. It doesn’t go crazy trying to find them so make sure all of your annotated methods are public.
2. Transaction boundaries are only created when properly annotated (see above) methods are called through a Spring proxy. This means that you need to call your annotated method directly through an @Autowired bean or the transaction will never start. If you call a method on an @Autowired bean that isn’t annotated which itself calls a public method that is annotated YOUR ANNOTATION IS IGNORED. This is because Spring AOP is only checking annotations when it first enters the @Autowired code.
3. Never blindly trust that your @Transactional annotations are actually creating transaction boundaries. When in doubt test whether a transaction really is active (see below)

My first problem was that the code was annotated improperly like this:

/**
 * This code example is BAD code, do not use it!
 */
class NonWorkingMyClass {

  @Autowired
  protected MyService myService;

  public void calledFirst() {
    // Do some setup work...

    // Call our internal method
    calledSecond();
  }

  @Transactional
  private void calledSecond() {
    MyObject myObject = myService.retrieveLatest();

    // Update some object fields
    myObject.setName("New Name");
  }
}

In this case someone would call NonWorkingMyClass.calledFirst(), it would then call calledSecond() and try to update the name field. This works if your XML configuration is set up properly but it will not be in a transaction. This can cause concurrency issues that won’t show up until it’s really inconvenient.

Here’s the working version of that code:

/**
 * This code example works
 */
class WorkingMyClass {

  @Autowired
  protected MyService myService;

  @Transactional
  public void calledFirst() {
    // Do some setup work...

    // Call our internal method
    calledSecond();
  }

  private void calledSecond() {
    MyObject myObject = myService.retrieveLatest();

    // Update some object fields
    myObject.setName("New Name");
  }
}

Now when someone called WorkingMyClass.calledFirst() it would do what you expect in a transaction and the transaction boundaries are properly respected.

This looks like a simple fix that should only take a few minutes but finding out that was the problem involved turning on lots of Spring DEBUG level logging, Googling, and actually testing to make sure the transactions were active. Before I knew what I know now I used some code from the first site I listed to show if I was inside a transaction or not. I was shocked and relieved when it showed that I wasn’t because it meant the concurrency issues weren’t due to bad programming, just bad configuration. Here are the methods that I came up with that you can use to see if you are in a transaction and even force your code to throw an exception if it isn’t. This can be invaluable if someone messes up an annotation in the future or breaks your XML configuration.

This code belongs in a utility class that is accessible from anywhere. There are two flags you will need to put somewhere:

transactionDebugging – Indicates we should do the transaction tests
verboseTransactionDebugging – Indicates we should print debug messages with the transaction tests

verboseTransactionDebugging has no effect if transactionDebugging is false.

class DebugUtils {
	private static final transactionDebugging = true;
	private static final verboseTransactionDebugging = true;

	public static void showTransactionStatus(String message) {
		System.out.println(((transactionActive()) ? "[+] " : "[-] ") + message);
	}

	// Some guidance from: http://java.dzone.com/articles/monitoring-declarative-transac?page=0,1
	public static boolean transactionActive() {
		try {
			ClassLoader contextClassLoader = Thread.currentThread().getContextClassLoader();
			Class tsmClass = contextClassLoader.loadClass("org.springframework.transaction.support.TransactionSynchronizationManager");
			Boolean isActive = (Boolean) tsmClass.getMethod("isActualTransactionActive", null).invoke(null, null);

			return isActive;
		} catch (ClassNotFoundException e) {
			e.printStackTrace();
		} catch (IllegalArgumentException e) {
			e.printStackTrace();
		} catch (SecurityException e) {
			e.printStackTrace();
		} catch (IllegalAccessException e) {
			e.printStackTrace();
		} catch (InvocationTargetException e) {
			e.printStackTrace();
		} catch (NoSuchMethodException e) {
			e.printStackTrace();
		}

		// If we got here it means there was an exception
		throw new IllegalStateException("ServerUtils.transactionActive was unable to complete properly");
	}

	public static void transactionRequired(String message) {
		// Are we debugging transactions?
		if (!transactionDebugging) {
			// No, just return
			return;
		}

		// Are we doing verbose transaction debugging?
		if (verboseTransactionDebugging) {
			// Yes, show the status before we get to the possibility of throwing an exception
			showTransactionStatus(message);
		}

		// Is there a transaction active?
		if (!transactionActive()) {
			// No, throw an exception
			throw new IllegalStateException("Transaction required but not active [" + message + "]");
		}
	}
}

In our previous code example we could use these new methods like this:

/**
 * This code example works
 */
class WorkingMyClass {

  @Autowired
  protected MyService myService;

  @Transactional
  public void calledFirst() {
    // Make sure we're using transactions.  Include the name of the class and method
    //   so it is easier to track down later if there are problems.
    DebugUtils.transactionRequired("WorkingMyClass.calledFirst");

    // Do some setup work...

    // Call our internal method
    calledSecond();
  }

  private void calledSecond() {
    MyObject myObject = myService.retrieveLatest();

    // Update some object fields
    myObject.setName("New Name");
  }
}

That’s it. Post in the comments if this helps you out or if you want to add to the code.

1. Open the start menu
2. Type “cmd”
3. Right click on “cmd” and select “Run as administrator”
4. Run subst like this:

   subst DRIVE: LOCATION

   DRIVE: will need to be a free drive letter like “F:”, “G:”, etc
   LOCATION will need to be the UNC path to your network share like this “\\myothercomputer\pictures\”
   Don’t forget to include the quotes if your LOCATION has spaces in it!

5. Restart Media Manager and try to add the new virtual drive to it, it should start working right away

You may need to do this on each reboot. I never reboot this computer so I haven’t tested it yet. You can put these commands in a batch file to make your life easier but you’ll need to make sure the batch file runs as an administrator.

Let me know in the comments if it works for you or not. If not I can probably help work out any kinks with you.

'xterm': unknown terminal type.

This is because either you haven’t installed ncurses-term (unlikely) or a symlink from /lib/terminfo/x/xterm to /usr/share/terminfo/x/xterm is missing. To cover all possibilities do this:

sudo apt-get install ncurses-term
sudo ln -s /lib/terminfo/x/xterm /usr/share/terminfo/x/xterm

Poof, your terminal works again!

Keep in mind that my illustration of how easy it was to write this driver is not a reflection on how easy it is to write all drivers. Some drivers take a ton of work. Make sure you read the documentation before you buy something so you don’t get stuck with some hardware you can’t use.

My first step was to find the documentation for the protocol for this device. I then scanned around to find the “Communication Protocol” and started digging. What this showed me is that there are two connections to support the protocol for this device. The first connection is called “CIN” for clock input and the second connection is called “DIN” for data input. Simple enough, especially if we’re using the standard Grove base shield and connectors. Just hook it up and make sure you keep track of which port you’re using and you’re ready to start programming. I used header 6 on my base shield so the relevant pins for me were D6 and D7. D6 was CIN and D7 was DIN.

Now you’ll see that there are six well defined bullet points explaining the basics of the protocol:

• Data needs to be ready before CIN, and DIN gets into the buffer on the rising edge of CIN.
• First 32 bits ’0′ are Start Frame
• Flag bit is two ’1′
• Calibration bits B7′,B6′;G7′,G6′ and R7′,R6′ are inverse codes of B7,B6;G7,G6 and R7,R6
• Gray data MSB first, and the order is BLUE, GREEN, and RED
• After all nodes data sent, need to seed another 32 bits ’0′ to update the data

Let’s step through these one by one to figure out how to send data to this device.

They first tell us that “data needs to be ready before CIN, and DIN gets into the buffer on the rising edge of CIN”. What this really translates to for you when programming is that when you want to send a bit to the device you should set that bit on the DIN pin (either 1 or 0), then set the CIN pin to high, and then set the CIN pin back to low. Now you’ve sent one bit of data to the device. Abstraction will make it so we can do this thinking once and then fall back on it later so let’s write a function that sends one bit:

private void sendBit(bool bit)
{
    // Get DIN into the proper state
    din.Write(bit);

    // Set the clock high
    cin.Write(true);

    // Set the clock low
    cin.Write(false);
}

This function makes the assumption that you’ve defined cin and din elsewhere. The setup for them in my case (using pins D6 and D7 as I described above) would look like this:

// Use D6 for CIN and D7 for DIN (Grove Base Shield v1.2 header #6)
OutputPort cin = new OutputPort(Pins.GPIO_PIN_D6, false);
OutputPort din = new OutputPort(Pins.GPIO_PIN_D7, false);

Now they tell us that the first 32 bits are all zeroes and that this is called a start frame. This makes me think it would be a good idea to expand our abstraction to let us send bytes and then write another function that would send this start frame. That would look like this:

private void sendByte(byte data)
{
    // Send the bits MSB first
    sendBit((data & 0x80) == 0x80);
    sendBit((data & 0x40) == 0x40);
    sendBit((data & 0x20) == 0x20);
    sendBit((data & 0x10) == 0x10);
    sendBit((data & 0x08) == 0x08);
    sendBit((data & 0x04) == 0x04);
    sendBit((data & 0x02) == 0x02);
    sendBit((data & 0x01) == 0x01);
}

private void sendStartFrame()
{
    // The start frame is 32 bits of zeroes
    sendByte(0);
    sendByte(0);
    sendByte(0);
    sendByte(0);
}

In the sendByte function I’m taking a byte and using logical AND and equals to extract the bits one by one. One of the next bullet points says the data is MSB first so we want to get the most significant (ie. largest value holding) bits first so that’s how I went about sending the bits. Now that we can send bytes sending the start frame is as easy as calling that function four times with the value 0.

Next they talk about flag bits. In the protocol it shows that after the start frame there are some flag bits. This tells us the two flag bits are both ones. Here’s a simple function that can do that:

private void sendFlagBits()
{
    // The flag bits are two 1s
    sendBit(true);
    sendBit(true);
}

Now this part gets a bit trickier but not too bad. They tell us that we need to send the inverse of B7, B6, G7, G6, R7, R6, followed by the actual color data itself as bytes. B7 and B6 are the two highest bits in the blue color component, G7 and G6 are the two highest bits in the green color component, and R7 and R6 are the two highest bits in the red color component. Sending that data with the functions we built up now is really easy.

private void sendColorData(byte red, byte green, byte blue)
{
    // Send the inverse bits of the B7, B6, G7, G6, R7, R6
    sendBit((blue & 0x80) != 0x80);
    sendBit((blue & 0x40) != 0x40);
    sendBit((green & 0x80) != 0x80);
    sendBit((green & 0x40) != 0x40);
    sendBit((red & 0x80) != 0x80);
    sendBit((red & 0x40) != 0x40);

    // Send the actual colors
    sendByte((byte)blue);
    sendByte((byte)green);
    sendByte((byte)red);
}

We’re almost there, there’s only one step left! Now we need to send the end frame. It turns out that the end frame is the same as the start frame but to keep the code readable I did this:

private void sendEndFrame()
{
    // The end frame is the same as the start frame
    sendStartFrame();
}

Now you have enough information to send a color to your device. We should probably wrap it up so that we can make it even easier to use though. Let’s just think about how this is going to be used in practice. A typical user will have a few of these LEDs chained together but for testing you might want to use just one. We know that the protocol requires a start frame, then flag bits, then color data, and the end frame if we use a single LED but for two LEDs it looks like this:

• Send start frame
• Send flag bits
• Send color data
• Send flag bits
• Send color data
• Send end frame

So for our first LED we want to send the start frame, the flag bits and the color data. For our last LED we want to send flag bits, the color data, and the end frame. Here’s a function that does that:

private void setColor(byte red, byte green, byte blue, bool first, bool last)
{
    // Is this the first color?
    if (first)
    {
        // Yes, send the start frame
        sendStartFrame();
    }
    else
    {
        // No, do nothing
    }

    // Send the flag bits
    sendFlagBits();

    // Send the colors
    sendColorData(red, green, blue);

    // Is this the last color?
    if (last)
    {
        // Yes, send the end frame
        sendEndFrame();
    }
    else
    {
        // No, do nothing
    }
}

The extra else blocks have no impact on the executable generated so they’re just there for clarity. You can remove them if you want. Now if you want to send a bunch of colors to a string of three LEDs you can do this:

setColor(255, 0, 0, true, false);
setColor(0, 255, 0, false, false);
setColor(0, 0, 255, false, true);

That would set a string of three LEDs to solid red, solid green, and solid blue. That’s it, your driver is written!

Check out my driver on Github to see a few more enhancements I added. My code has an abstraction of a color from three integers into an RGB object so it’s easier to pass around and also has a function that can set a string of LEDs from an array of RGB objects. There’s some sample code as well and if you want to see the system in action check out these simple videos:

• Cycling random colors
• Cycling red, green, and blue

Post in the comments and share your thoughts and project ideas. If you use this library please let me know!

Plugin execution not covered by lifecycle configuration

I checked the usual places but didn’t find a solution to the issue. Then I decided to try adding the m2e plugin from this update site:

http://download.eclipse.org/technology/m2e/releases

After adding the plugin and restarting my IDE I got two different error messages:

maven-dependency-plugin (goals "copy-dependencies", "unpack") is not supported by m2e.
Project configuration is not up-to-date with pom.xml. Run project configuration update.

The second error had a quick fix so I tried that and it worked. Now the Java example application that uses the Play framework and the one that uses Spring MVC and Hibernate both work. However, the ones that used JAX-RS and embedded Jetty did not. They still showed the maven-dependency-plugin error. The fix is to add the following XML in the build section of your pom.xml:

<pluginManagement>
	<plugins>
		<plugin>
			<groupId>org.eclipse.m2e</groupId>
				<artifactId>lifecycle-mapping</artifactId>
				<version>1.0.0</version>
				<configuration>
					<lifecycleMappingMetadata>
						<pluginExecutions>
							<pluginExecution>
								<pluginExecutionFilter>
									<groupId>org.apache.maven.plugins</groupId>
									<artifactId>maven-dependency-plugin</artifactId>
									<versionRange>[1.0.0,)</versionRange>
									<goals>
										<goal>copy-dependencies</goal>
									</goals>
								</pluginExecutionFilter>
								<action>
									<ignore />
								</action>
							</pluginExecution>
						</pluginExecutions>
					</lifecycleMappingMetadata>
				</configuration>
			</plugin>
		</plugins>
	</pluginManagement>

After that you’ll have to do the quick fix for the error “Project configuration is not up-to-date” again and then you’ll be error free, at least in your pom.xml…

Post in the comments and let me know if it worked or if you need any help.

     [exec] 12/03/21 22:51:47 INFO mapred.JobClient: Task Id : attempt_201203212250_0001_m_000002_1, Status : FAILED
     [exec] Error initializing attempt_201203212250_0001_m_000002_1:
     [exec] ENOENT: No such file or directory
     [exec] 	at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
     [exec] 	at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:521)
     [exec] 	at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
     [exec] 	at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240)
     [exec] 	at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:216)
     [exec] 	at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1352)
     [exec] 	at java.security.AccessController.doPrivileged(Native Method)
     [exec] 	at javax.security.auth.Subject.doAs(Subject.java:416)
     [exec] 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
     [exec] 	at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1327)
     [exec] 	at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1242)
     [exec] 	at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2541)
     [exec] 	at org.apac

It wasn’t immediately apparent to me what file wasn’t found from the error messages so I checked the logs, the JobTracker, my code, ran some known good jobs that also failed, basically everything I could think of. It turns out that due to me accidentally running a script as “root” (don’t worry, it was only on my desktop) that the permissions of several files in the hdfs user’s home directory had changed ownership to “root”. Because of that Hadoop was unable to create files in the /usr/lib/hadoop-0.20 directory.

NOTE: These steps assume you are using Hadoop 0.20. Adjust the paths in the commands accordingly if you aren’t.

If you want a quick fix try these steps (only if you take full responsibility for anything that may go wrong):

1. Stop Hadoop using the stop-all.sh script as the hdfs user
2. su to the hdfs user
3. Run this:

   chown -R hdfs:hdfs /usr/lib/hadoop-0.20 /var/*/hadoop-0.20

4. Restart Hadoop using the start-all.sh script as the hdfs user

Now your jobs should start running again. Post in the comments if this procedure works for you or if you need any help.

First, you’ll want to add the EPIC (Eclipse Perl Integration Component) as described on the EPIC site. That will add support for creating Perl projects, syntax highlighting, and all that.

Next, set a breakpoint in one of your Perl scripts and try to debug it. If you’re unlucky you may get one of two error messages. One error message wants you to install PadWalker which is a Perl module that handles all of the debugging niceties for Eclipse. To install that you can either use CPAN or apt. Using apt is as simple as:

sudo apt-get install libpadwalker-perl

Once you install PadWalker restart Perl and try to debug one of your scripts again. If it works, you’re set. The second possible error message is below…

Now, you’ve come all this way and it still doesn’t work. You’ve probably received an error message like this:

Could not create the view: Plug-in "org.eclipse.debug.ui" was unable to instantiate class "org.eclipse.debug.internal.ui.views.variables.VariablesView".

If you dig deeper you’ll see errors like this:

java.lang.ClassCircularityError: org/eclipse/debug/internal/ui/DebugUIPlugin

And if you dig even deeper you’ll see errors like this:

Conflict for 'org.epic.perleditor.commands.clearMarker'

The fix for this was tricky to figure out so just follow these steps:

1. Close Eclipse
2. Uninstall libpadwalker-perl by running

   sudo apt-get remove --purge libpadwalker-perl

3. Restart Eclipse and try to set a breakpoint in a Perl script, it should fail (no breakpoint should appear)
4. Close Eclipse
5. Reinstall libpadwalker-perl by running

   sudo apt-get install libpadwalker-perl

6. Restart Eclipse, set a breakpoint, and start debugging again

At this point the variables and breakpoints should always work. Unfortunately the expressions panel will not. It looks like this is not supported in EPIC just yet. But, in any case, you now have a full fledged Perl debugger so you can (mostly) stop using print statements to debug your code post mortem.

There are some quirks to note:

1. “Step Over” (typically F6) does not work as expected and will step into modules. If “Step Return” worked this wouldn’t be a problem but it doesn’t (see the next bullet point). In this case if you are trying to step over a module you may have to back out and set a breakpoint where the execution will return to the script you want to debug.
2. “Step Return” (typically F7) does not work as expected. It will usually run until your script ends or hits a breakpoint.
3. The console window will not let you run arbitrary Perl code so it’s not a simple replacement for the expressions panel
4. Perl modules (files with a .pm extension) may not appear with syntax highlighting enabled. If you are debugging Perl modules you may want to retool your setup and run the module as a Perl script OR have Perl load your module from a file with a .pl extension.

Good luck. Now clean up/fix that Perl code and post in the comments.

Specifically I was trying to use assertThat() and is() but didn’t know where to find them. After a bit of Googling I found the two static imports that I needed to copy the code without qualifying assertThat() as Assert.assertThat() and the same goes for is(). They are:

import static org.hamcrest.CoreMatchers.is;
import static org.junit.Assert.assertThat;

I have to admit that org.hamcrest is a bit less obvious than I would have liked.

What you’ll need to do first is decide what you want to do in the processing thread. This code will go into a Java Runnable like this:

Runnable runnable = new Runnable() {
	@Override
	public void run() {
		// Do something exciting here
	}
};

Next you’ll need to decide how many threads you want to run. If you wanted to run four threads you could do this:

int threadCount = 4;

for (int threadLoop = 0; threadLoop < threadCount; threadLoop++) {
	// XXX - Put the runnable block from above right here

	// Create a new thread
	Thread thread = new Thread(runnable);

	// Add the thread to our thread list
	threads.add(thread);

	// Start the thread
	thread.start();
}

That will start four threads. It’s best to use a variable so you can update it and use it in other places like calculating your statistics. Now let’s wait for all the threads to finish:

// Loop through the threads
for (Thread thread : threads) {
	try {
		// Wait for this thread to die
		thread.join();
	} catch (InterruptedException e) {
		// Ignore this but print a stack trace
		e.printStackTrace();
	}
}

Finally, you’ll want to time all of this. I do something very simple here. Before all of the code I do this:

long startTime = new Date().getTime();

After all of the code I do this:

long endTime = new Date().getTime();
long durationInMilliseconds = endTime - startTime;

With all of that in place you can now measure how long your code ran and then calculate important metrics about it. For example, if this code did 10,000 operations per thread and ran with 4 threads you would then take the duration and divide that by 40,000 and you’d get an idea of how many milliseconds it took per operation. Just make sure you use doubles or you’ll lose all of your precision due to coercion. Do this (assuming that your number of operations is stored in a variable called “operations”):

double millisecondsPerOperation = (double) durationInMilliseconds / (double) operations;
double operationsPerMillisecond = (double) operations / (double) durationInMilliseconds;

These are just reciprocals of each other but sometimes one value is a lot easier to understand than the other so I usually calculate them both.

Now that you have those statistics you can try different thread counts, optimize code/loops, etc. Good luck! Post in the comments with any ideas and/or issues.