Recording and controlling the 4D light field in a microscope

JournalofMicroscopy,Vol.235,Pt22009,pp.144–162

Received7November2008;accepted7April2009

Recordingandcontrollingthe4Dlightfieldinamicroscope

usingmicrolensarrays

M.LEVOY∗,Z.ZHANG∗&I.MCDOWALL†∗ComputerScienceDepartment,StanfordUniversity,Stanford,California,U.S.A.†FakespaceLabs,241PolarisAve.,MountainView,California,U.S.A.

Keywords.Illumination,lightfield,lightfieldmicroscope,microlensarray,

plenopticfunction,syntheticapertureimaging,spatiallightmodulator,

sphericalaberration,Shack–Hartmannsensor.

Summary

Byinsertingamicrolensarrayattheintermediateimageplane

ofanopticalmicroscope,onecanrecordfour-dimensional

lightfieldsofbiologicalspecimensinasinglesnapshot.Unlike

aconventionalphotograph,lightfieldspermitmanipulation

ofviewpointandfocusafterthesnapshothasbeentaken,

subjecttotheresolutionofthecameraandthediffraction

limitoftheopticalsystem.Byinsertingasecondmicrolens

arrayandvideoprojectorintothemicroscope’sillumination

path,onecancontroltheincidentlightfieldfallingonthe

specimeninasimilarway.Inthispaper,wedescribea

prototypesystemwehavebuiltthatimplementstheseideas,

andwedemonstratetwoapplicationsforit:simulatingexotic

microscopeilluminationmodalitiesandcorrectingforoptical

aberrationsdigitally.

Introduction

Thelightfieldisafourdimensional(4D)functionrepresenting

radiancealongraysasafunctionofpositionanddirection

inspace.Overthepast10yearsourgrouphasbuiltseveral

devicesforcapturinglightfields(Levoy,2000;Levoy,2005;

Ng,2005b;Wilburnetal.,2002).Inparticular,Ngetal.isa

handheldcamerainwhichamicrolensarrayhasbeeninserted

betweenthesensorandmainlens.Aphotographtakenby

thiscameracontainsagridofcircularsubimages,oneper

microlens.Eachsubimagerecordsonepointinthescene,and

withinasubimageeachpixelrecordsonedirectionofview

ofthatpoint.Thus,eachpixelinthephotographrecordsthe

radiancealongonerayinthelightfield.

Recently,wehaveshownthatbyinsertingamicrolensarray

attheintermediateimageplaneofanopticalmicroscope,

wecancapturelightfieldsofbiologicalspecimensinthe

Correspondenceto:MarcLevoy.Tel:+16507254089;fax:+16507230033;e-mail:levoy@cs.stanford.edusameway.Wecallthisalightfieldmicroscope(LFM)(Levoy,

2006).Fromtheimagecapturedbythisdevice,onecan

employlightfieldrendering(Levoy&Hanrahan,1996)to

generateobliqueorthographicviewsorperspectiveviews,at

leastuptotheangularlimitofraysthathavebeencaptured.

Sincemicroscopesnormallyrecordorthographicimagery,

perspectiveviewsrepresentanewwaytolookatspecimens.

Figure1showsthreesuchviewscomputedfromalightfield

ofmouseintestinevilli.

Startingfromacapturedlightfield,onecanalternativelyuse

syntheticaperturephotography(Isaksenetal.,2000;Levoy

etal.,2004)toproduceviewsfocusedatdifferentdepths.

Twosuchviews,computedfromalightfieldofGolgi-stained

ratbrain,areshowninFig.10.Theabilitytocreatefocal

stacksfromasingleinputimageallowsmovingorlight-

sensitivespecimenstoberecorded.Finally,byapplying3D

deconvolutiontothesefocalstacks(Agard,1984),onecan

reconstructastackofcross-sections,whichcanbevisualized

usingvolumerendering(Levoy,1988).

Summarizing,theLFMallowsustocapturethe3Dstructure

ofmicroscopicobjectsinasinglesnapshot(andthereforeat

asingleinstantintime).Thesacrificewemaketoobtain

thiscapabilityisareductioninimagesize.Specifically,if

eachmicrolenssubimagecontainsN×Npixels,thenour

computedimageswillcontainN2fewerpixelsthanifthe

microlenseswerenotpresent.Inreturn,wecancomputeN2

uniqueobliqueviewsofthespecimen,andwecangeneratea

focalstackcontainingNsliceswithnon-overlappingdepthsof

field(Levoy,2006).Notethatthistrade-offcannotbeavoided

merelybyemployingasensorwithmorepixels,because

diffractionplacesanupperlimitontheproductofspatialand

angularbandwidthforagivenaperturesizeandwavelength,

regardlessofsensorresolution.Despitethislimit,lightfields

containmuchusefulinformationthatislostwhenanobjectis

photographedwithanordinarymicroscope.

Whiletechnologiesforrecordinglightfieldshaveexisted

formorethanacentury,technologiesforgeneratinglight

C󰀊2009TheAuthorsJournalcompilationC󰀊2009TheRoyalMicroscopicalSocietyRECORDINGANDCONTROLLINGTHE4DLIGHTFIELD145

Fig.1.Threeobliqueorthographicviewsfromtheindicateddirections,computedfromalightfieldcapturedbyourLFM.ThespecimenisafixedwholemountshowingL.monocytogenesbacteriainamouseintestinevillus7hpostinfection.Thebacteriaareexpressinggreenfluorescentprotein,andtheactinfilamentsinthebrushborderofthecellscoveringthesurfaceofthevillusarelabelledwithrhodamine-phalloidin.Scalebaris10μm.Imagingemployeda60×/1.4NAoilobjectiveandanf/30microlensarray(seesection‘Prototypelightfieldilluminatorandmicroscope’).ContrastofthebacteriawasenhancedbyspatiallymaskingtheilluminationasdescribedinFig.7.Theinsetatleftshowsacropfromthelightfield,correspondingtothesquareinthefirstobliqueview.Inthisinset,weseethecircularsubimagesformedbyeachmicrolens.Theobliqueviewswerecomputedbyextractingonepixelfromeachsubimage–apixelnearthebottomofeachsubimagetoproducetheleftmostviewandapixelnearthetoptoproducetherightmostview.