Category: Uncategorized

It has been the depth of Pittsburgh winter her recently with snow on the ground for several weeks in a row and weather in the teens. Since it rained and then froze, everything is icy too. Yesterday I spent a good hour shoveling the sidewalk and cars to scrape the ice off of everything. Today dawned cold but sunny and the whole day has been super sunny. The greenhouse has been an incredible joy with the warmth of the sun with the cool air as the vents let in a refreshing breeze. I even threw a pillow on the ground and took a snooze amidst the plants. I’m hopeful that the plants can be starting to recover as we get more sunny days and start edging our way towards March, where it really does start getting quite a bit more sunny. I thought it would be worth highlighting one of the plants in the greenhouse that has done the best over the years, consistently doing well: the purple queen. It’s surprisingly immune to almost anything and just keeps trucking and being awesomely purple. It gets a bit leggy and needs some pruning but just wanted to say congrats purple queen, you doin’ great!

So almost two years ago I started an experiment to see how well Micro Dwarf Tomatoes could be grown indoors and outdoors and which varieties produced and tasted the best: 2020.05.08: Micro Dwarf Tomato Experiment, Day 0. It turns out micro dwarf tomatoes can do spectacularly well indoors because here it is 642 days later (nearly 2 years) and I’m still harvesting lots of tomatoes! I think only a few of the 10 original varieties I planted are still around and kicking and we’ve been through two spider mite and one white fly infestations (resolved with the help of beneficial insects).

The coolest thing about these tomatoes is I haven’t even really tried to keep them alive, they just kinda sit in my office. I’ve mostly just kept watering them, periodically throwing some Miracle Gro, and harvesting tomatoes when they become ripe. I’ve been super impressed!

A few of my tomatoes and peppers have germinated. I didn’t have great germination rates, possibly because it was too damp and some algae started growing. But that’s OK, two tomatoes and a pepper should get me started in 4-6 weeks when the sun starts returning and the greenhouse starts warming.

The weather has been quite cold outside, it’s been below 10 F for three days in the past week and some days haven’t gotten above 20 F. The greenhouse has been doing pretty well temperature-wise, it hasn’t been below 60 F and on one of the sunny days even made it up to low 80s!

It’s 2022! Pretty crazy, and winter is hitting full force here in Pittsburgh. We’ve had single digit lows and snow on the ground for a week now. I overwatered the greenhouse and with the cold temps and not much light (even with grow lights), most everything is doing a bit poorly – lots of yellowing and leave loss. Hopefully things will turn around in a couple of months when spring starts arriving. I did up the temperature in my greenhouse a few degrees from 65-70 range to 68-73 F because it always felt a little too chilly to me. But to take advantage of those cooler temperatures, I planed a bunch of cool weather crops: romaine lettuce, sugar snap peas, bok choy, mustard, and cilantro. They are just starting to come up! Hopefully I’ll be able to get a harvest before the increased sun in March spikes the greenhouse temperatures.

I also was doing some research on new micro tomatoes and discovered a few new varieties to try: Siam, Red Velvet, Cocoa, and Heartbreaker Twiggy. On the pepper side, I’ll also try pot-a-peno and fresh bites, both dwarfs.

Ahhh!!! My journal paper with a 14 page limit was 16.5 pages this morning and after an entire day of chop chop chopping, I have reduced it by to 15 pages. One more page to go (somehow). In the mean time, I just packed to leave in 4 hours when I head to the airport and fly back to Pittsburgh. Fun fun!

Four years of grinding work in graduate school, done with classes, put out some conference papers, published a journal paper, and people keep asking when I’m going to be done. It must be that time in the PhD program to propose a thesis. Next Monday I’m giving my oral proposal, but I just mailed the thesis document to my thesis committee members and the Robotics Institute in general. The details are:

Vision-Based Control of a Handheld Micromanipulator for Robot-Assisted Retinal Surgery

Abstract – Surgeons increasingly need to perform complex operations on extremely small anatomy. Many promising new surgeries are effective, but difficult or impossible to perform because humans lack the extraordinary control required at sub-mm scales. Using micromanipulators, surgeons gain better positioning accuracy and additional dexterity as the instrument smoothes tremor and scales hand motions. While these aids are advantageous, they do not actively consider the goals or intentions of the operator and thus cannot provide context-specific behaviors, such as motion scaling around anatomical targets, prevention of unwanted contact with pre-defined tissue areas, and other helpful task-dependent actions.

This thesis explores the fusion of visual information with micromanipulator control and builds a framework of task-specific behaviors that respond synergistically with surgeon’s intentions and motions throughout surgical procedures. By exploiting real-time microscope view observations, a-priori knowledge of surgical procedures, and pre-operative data used by the surgeon while preparing for the surgery, we hypothesize that the micromanipulator can better understand the goals of a given procedure and deploy individualized aids in addition to tremor suppression to further help the surgeon. Specifically, we propose a vision-based control framework of modular virtual fixtures for handheld micromanipulator robots. Virtual fixtures include constraints such as “maintain tip position”, “avoid these areas”, “follow a trajectory”, and “keep an orientation” whose parameters are derived from visual information, either pre-operatively or in real-time, and are enforced by the control system. Combining individual modules allows for complex task-specific behaviors that monitor the surgeon’s actions relative to the anatomy and react appropriately to cooperatively accomplish the surgical procedure.

Particular focus is given to vitreoretinal surgery as a testbed for vision-based control because several new and promising surgical techniques in the eye depend on fine manipulations of delicate retinal structures. Preliminary experiments with Micron, the micromanipulator developed in our lab, demonstrate that vision-based control can improve accuracy and increase usability for difficult retinal operations, such as laser photocoagulation and vessel cannulation. An initial framework for virtual fixtures has been developed and shown to significantly reduce error in synthetic tests if the structure of the surgeon’s motions is known. Proposed work includes formalizing the virtual fixtures framework, incorporating elements from model predictive control, improving 3D vision imaging of retinal structures, and conducting experiments with an experienced retinal surgeon. Results from experiments with ex vivo and in vivo tissue for selected retinal surgical procedures will validate our approach.

Thesis Committee Members:
Cameron N. Riviere, Chair
George A. Kantor
George D. Stetten
Gregory D. Hager, Johns Hopkins University

A copy of the thesis proposal document is available at:
http://briancbecker.com/thesis/becker_proposal.pdf

After making prison loaf (e.g. a nutritionally complete bread similar to meatloaf but made out of vegetables for prison inmates that really quite honestly tastes like nothing), we decided that perhaps maybe a better idea was to combine lots of different delicious desserty things. Somehow this morphed into dessert burritos. Yeah don’t look at me – I don’t from which bush that idea sprang behind from and tackled us, but it seemed like a pretty good idea at the time. Maybe it was the horrid taste of nothingness left behind from the insipid prison loaf that persists for months regardless of how often you try to scrub the memory from your mind. Or maybe not. Anyhow, thus began the trip to  buy goodies and make brownies and prepare for something possibly great. Recipes for dessert burritos were a bit hard to come by (OK, I lie – I didn’t even check) so I just decided to wing it. Here is the recipe to our culinary masterpiece:

1) Start with one soft taco burrito thingie from the store.

2) Place on brownie square in the middle of the burrito.

3) Spread Nutella on the burrito. If you can somehow make it not look like a mud pie, do so.

4) Scoop one vanilla and one chocolate ice cream ball onto the burrito.

5) Liberally sprinkle strawberries and blueberries or other fantastic fruit on your burrito.

6) No dessert is complete without drizzled chocolate syrup, so go hog wild.

7) Fold your burrito, making sure to fold the bottom in so it doesn’t leak too too much. Notice my ultra-classy paper plate.

8 ) Eat your dessert burrito! Please don’t look as scared as I am at the prospect.

So yeah there you have it. Dessert burritos. I’ve got to admit, the burrito was a bit saltier than I was expecting, so if you can procure non-salted ones, it would be in your best interest. I think I might just recommend a regular old sundae unless you are looking for something particularly unique.

Say your baseball hitting robot is using cameras to track and hit a thrown ball. In this and other systems where computer vision is a sensor for time-critical processes such as a tight control loop, latency is very important. My project, Micron, runs a high-frequency control loop to achieve micromanipulations in surgical environments. It runs with the same principle of games from alot. Micron depends on cameras to know where important anatomy is, track the tool tip, and register the cameras to other high-bandwidth position sensors. This week I analyzed and tried to optimize camera performance.

My system has dual Flea2 PointGrey cameras that can capture at 800×600 in YUV422 @ 30Hz, which was then converted to RGB on the PC. In order to increase bandwidth, I first swapped to Format7 Raw8 mode. The custom Format7 mode has a number of advantage:

• Bayer. You get the raw values from the CCD in Bayer format. This is a more compact (and native) representation than RGB or a YUV encoding, allowing for more data to pass over Firewire to the PC. Also, fast de-Bayering algorithms exist in popular software like OpenCV’s cvtColor or in GPU. If you are playing Fortnite Hacken game on your computer check out this iGaming Software Provider, it will help make your games run smoother. You can specify a custom region of interest so you don’t have to transfer the entire frame. This allows you to choose the ideal point in the “frame size vs. frame rate” curve. If you only need part of the frame, you can increase how fast images are being transferred to the PC.
• Event notifications. Technically also available in other modes as well, event notifications allow you to process partial frames as they arrive. Each image consists of individual packets that are transferred sequentially over Firewire and the driver will let you know when each packet arrives. Theoretically, if your image processing algorithm operates on the image sequentially, you can transfer and process in tandem. For instance, at full 1032×776 resolution, it takes a Flea2 ~33 ms from the first image packet to the last image packet. Traditionally you can’t start processing the image until the last packet has arrived and the full image is available. However, if you are say doing color blob tracking where you need to test each pixel to see if it is some predefined color such as red, you can start as soon as parts of the image arrive. After 10% of the image arrives, you can image process that part while the second 10% is being transferred from the camera. This can theoretically significantly reduce overall latency because image transfer and image processing are no longer mutually exclusive.

I tried using all these tricks to reduce latency, although I wasn’t able to get event notifications properly. At first, it seems like a great idea. I was able to get up to 128 notifications per image, which is amazing! If my algorithm runs realtime, that means I could reduce my processing overhead to the time taken to process less than 1% of the image. However it seems they have a number of limitations and peculiarities that may be due to my poor programming or setup. A recently released game that is working with low latency is mu origin europe for Android, Find the best mod apk for this android game.
To test latency, I used the color detection and blob tracker code I run to find colored paint marks on my tool tip and applied to a blinking LED problem. At a random time, I turn a green LED on and I wait until my color blob detector sees the LED – the time between these two events is, by definition, the latency between when something happens and the cameras see it happening. The result is show below. Total is the time between turning the LED on and detecting it. Process time is the image processing time. Wait time is the average time a frame spends while waiting for the previous image to process. And shutter time is the average time between the LED turning on and the shutter for the next image capturing the LED.

Why are you so crazy, Linux? A friend asked me to look at why his installation of Matlab + code for the ImageNet 2010 competition on his VMWare Linux box wasn’t working. I logged on and typed in matlab, but got “matlab: command not found”, which was strange because he said he had installed it. Doing a “locate matlab” told me matlab was in /usr/bin/MATHWORKS_R2008B/bin so I executed

PATH=/usr/bin/MATHWORKS_R2008B/bin:$PATH

export PATH

matlab

That worked as I got a nice Matlab splashscreen, but then it just crashed printing out a cryptic “Opening log file: /home/usr/java.lgo.11195” message. That log file had the error “Could not reserve enough space for object heap”, which sounded suspiciously like not enough memory. Sure enough, only 384 MB of RAM were allocated to the VMWare virtual machine. Change that and reboot.

And of course by then, my local export PATH command was invalid so I added the following line to /etc/profile:

PATH=/usr/bin/MATHWORKS_R2008B/bin:/usr/bin/MATHWORKS_R2008B/bin/utils/mex;
^^^ DON'T DO THIS!!!

Reboot and I can’t even log in. Ctrl + Alt + F1 to get to command console and massive amounts of errors later I found out that bleck, I overwrote all the other PATH variables too. Now I have to /usr/bin/sudo /usr/bin/nano /etc/profile and change the line to include the old path as well.

PATH=$PATH:/usr/bin/MATHWORKS_R2008B/bin:/usr/bin/MATHWORKS_R2008B/bin/utils/mex

export PATH

Whew, now matlab starts up and we can execute mex. Except now matlab keeps whining about “cannot write to preference file “matlab.prf” in “/home/user/.matlab/R2008b/. A quick google search says we need to execute

sudo chown user /home/user/.matlab/R2008b/matlab.prf

sudo chmod a+w /home/user/.matlab -R

That first one might be redundant, but there are several files matlab needs to edit and the first command only solves the first problem.

Running the make file resulted in “/usr/bin/ld: cannot find-lstdc++” which is interesting. Not entirely sure what was going on, I decided to do a sanity check and wrote a nice hello_world.cpp program and tried compiling it with g++ only to discover g++ wasn’t installed. Install g++, do another sudo ldconfig for good measure, and bam it all compiles nicely. However, running make from the feature directory whined about u_int32_t in item.hpp, so a quick “typedef unsigned int u_int32_t;” in item.hpp later, g++ is complaining about not being able to link to -lvl. Turns out their Makefile needed to be pointed to ./3rd-party/vlfeat/bin/glx and then things are happy.

Whew! Finally, we can run their example script: extract_bow.sh. Alas still no luck, all sorts of complaining going on. The first is “./vldsift: error while loading shared libraries: libvl.so: cannot open shared object file: No such file or directory” Bleck! Adding the directory that libvl.so is in to /etc/ld.so.conf seemed to work and the demo program ran.

Sigh…Linux, why are you so crazy? Or maybe the better question is: you guys who programmed Linux, why are you so crazy?

So with all my JPEG decoder analysis, the library that performed the best was the libjpeg SIMD extension. It is several times faster than the normal libjpeg because it uses the modern SSE instructions for the core encoding/decoding routines. Not only does it use the SSE instruction set, but it does so in assembly language – not using compiler intrinsics. This means it is even faster because compiler intrinsics are notorious for producing inefficient or even wrong code. Unfortunately, it is only 32-bit (x86) – and trying to compile a 64-bit version of the library would mean porting over all the assembly language code.

At first glance, porting the assembly code from 32-bit to 64-bit seems intimidating, but after a while you realize that there are many versions of the same thing. Each routine is coded in MMX, 3Dnow, regular SSE, SSE2. And all these options can be applied to the slow integer, fast integer, and floating-point algorithms so you get dozens of assembly language files that you really just don’t need. The fastest combination is SSE2 and fast integer math. We can remove everything else because almost all recent processors in the past 5 years or so support SSE2 (Intel Pentium 4 and Intel Pentium M and AMD Athlon 64 and up). The fast integer math algorithms might cause a small reduction in image quality, but it’s not very noticeable and you get a solid 3% improvement in speed.  Disabling everything but SSE2 fast integer algorithms leaves you only with 10 assembly language files to modify, which isn’t too bad.

Now don’t get me wrong, as you can see from my previous post, converting from 32-bit to 64-bit assembly is a giant pain. It took me several days, putting in hours each day to carefully convert and debug each one to  make sure it was at least working with my data. I finally got it all seemingly working 64-bit (it doesn’t crash when loading some images off Facebook or from my camera), which is quite a good feeling because to my knowledge nobody else has done that.

I even mexed my port in a 64-bit Matlab mex function named readjpegfast. Unfortunately, Matlab stores its images all weird so a lot of time is wasted just re-arranging the pixel data into a column-first, RGB plane-separated format. For small images roughly 640×480, I get an impressive ~2X improvement on loading 2000 images over imread (Intel Core2 Duo 2.4 GHz T8300 laptop):

>> tic, for i = 0:2000, img = imread(sprintf(‘%0.4d.jpg’, i)); end, toc
Elapsed time is 21.120399 seconds.
>> tic, for i = 0:2000, img = readjpegfast(sprintf(‘%0.4d.jpg’, i)); end, toc
Elapsed time is 9.714863 seconds.

Larger images unfortunately don’t fair so well just because I have to do this format conversion (it would be better if I modified the library to load images into memory using the Matlab format, but that’s way too much work. The 7 megapixel pictures from my camera only saw about a 1.25X improvement:

>> tic, for i = 0:35, img = imread(sprintf(‘big\\%0.4d.jpg’, i)); end, toc
Elapsed time is 13.393228 seconds.
>> tic, for i = 0:35, img = readjpegfast(sprintf(‘big\\%0.4d.jpg’, i)); end, toc
Elapsed time is 10.068488 seconds.

Oh well, since I plan to be mostly using this in C++ where the default data-format of libjpeg is perfect for my uses, this is still a huge win. Soon I hope to be releasing a package that includes my 64-bit port of libjpeg SIMD.

It’s terrible…yeah you heard me. Assembly language is hard even at the best of times and it is all so hardware dependent that porting is a super pain in the neck. Even on the same processor, the change from 32-bit code to 64-bit code is annoying.

First rule of business, change BITS 32 to BITS 64. It seems obvious but when you have a bunch of *.asm files and you are doing them one by one, forgetting one can cause “error: impossible combination of address sizes” which will proceed to befuddle you for the next 10 minutes. Or not as it seems I’m the only one on Google has actually gotten this error.

I also found my first use for Wolfram Alpha: taking the modulus of 16-byte hex numbers to determine if they are 16-bit aligned. Yeah, I couldn’t even find a quick way to do that in Matlab, which is surprising. But then I realized I was being really braindead because it is really simple to see if a pointer address is 16-byte aligned: the last digit should be zero! Oops…I’m being silly again.

The errror “error LNK2017: ‘ADDR32’ relocation to ‘.rdata’ invalid without /LARGEADDRESSAWARE:NO” means you forgot to add “default rel” to the top of your assembly language file.

Apparently you also have to preserve some of the new registers across function calls. None of the NASM/YASM manuals or anything I read mentioned this! Code after I ran one of my functions was crashing and through turning off bits of the code, I was able to narrow it down to mov r12,ecx to store the first parameter. Of course then the thought struck me: maybe I need to preserve r12 so I finally had to Google “preserve r12 assembly.” I found some 64-bit sample assembly code from Mark William’s blog which had some comments about preserving r12-r15 and that seemed to fix the problem.

This weekend was Thanksgiving and I drove 8 hours from CMU in Pittsburgh, PA to my grandparent’s place in Newland, NC.  Well actually although I’d rather not admit it, the 8 hours turned into 11 hours because I read the map wrong. I can understand multiple roads merging and sharing the same physical road. For instance, one road going south and another going east can share a physical road going southeast for a while. However, a road designated as traveling south and a road claiming to go north should NOT be sharing! My directions said follow I-81 S/US-52 N – in what crazy system is going down this road both north and south?!? Very confusing it is, and caused me to travel an extra 150 miles.

Regardless of how many times I got lost (once! OK? No need to insult me!), I arrived the night before Thanksgiving. My grandparents have a Christmas tree farm and late every fall they cut, haul, bail, and load Christmas trees – some wholesale, some for their retail customers in Florida where my Uncle lives, and some for the Choose & Cut customers. Often the Choose & Cut customers are families who want to come up and have a good time looking around for a tree, have us cut and load it for them, and then go home with a frees tree they selected themselves. It is really quite nice to see families having fun with this – some spend hours analyzing what seems like every one of the thousands of trees to find the perfect one. However, this means the whole family works over Thanksgiving and the weekend. Which means I spend all day outside in freezing weather trying to process 6-14 foot Christmas trees. The first day it snowed on us while we were trying to bail 50 or so trees. My shoes got soaked, my jeans got soaked, everything. It was quite miserable, and worst of all it started snowing just as the sun was going down and the wind was picking up. By the next day the temperature had dropped to 15 F with winds 15-20 mph gusting up to 50 mph. It was quite bitter. But luckily my grandmother and mom were making cookies and hot apple cider for the customers who came so we could go to the barn to warm up and get some warm snacks. Fortunately, Sat and Sun warmed up significantly to 45 F or so without much wind, so I was able to work without being miserable. All and all, I think I helped cut, haul, bail, and load a few hundred trees. Wow, I’m tired just thinking back on it. We would always dread the monster ones – or sequoias as we called them – that were 12+ feet tall and weighed…well I’m not even sure other to say I couldn’t lift it myself. Which I’m embarrassed to say isn’t really saying all that much because I’m a skinny nerd. Suffice to say, I was very sore and tired and amazingly enough went to bed before midnight almost all the nights.

But it was a good time and I made some money out of it. It’s always good to help out the family biz – and I got some great meals out of it too. If you haven’t made up your holiday plans yet kensingtontours.com/tours/africa may be a great choice.