The right of Dennis Jackson to be1© 1992 D.Jacksonidentified as the author of thiswork has been asserted by him inThe Monday Morning SyndromeDennis JacksonJoint Network Team,Rutherford Appleton Laboratory,Chilton, Didcot,Oxfordshire OX11 0QX, UKTel: +44 (0)235 445724 Fax: +44 (0)235 446251D.Jackson@IntroductionIt is well underst ood t hat hackers and ot her comput er criminals operat e out side of normal office hours. This paper will present t he event s and subsequent repercussions of a weekend in April 1991. At t he t ime I was working at the University of London Computer Centre and had established a working relat ionship wit h several groups involved in t he invest igat ion of intrusions to computer systems.What is a Hacker?The term "hacker" has come to mean anyone who makes use of a data network to misuse a computer system. The term has acquired many meanings, including, a creative programmer, one who illicitly breaks into computers, a novice golfer who digs up the course, a taxi-cab driver, an inferior journalist, and ditch-digger. Confusion between the first two interpretations results in the perception that one need be brilliant or creative to break into computers. This may not be true. Many trouble-makers are patient and plodding. They use the same few techniques over and over again, but hardly show any creative brilliance in discovering new security flaws.Attempts have been made to use other words that better express the anti-social and unwanted actions of these people. Alternatives for describing someone who breaks into computers are: the English word "Cracker," and the Dutch term "Computerredebrenk" (literally, computer peace disturber). This paper will use the term hacker to denote someone who misuses computer systems.MondayThe Telephone CallOn a Monday morning in April 1991 I received a telephone call from a system manager at a medical research institute to the north of London. He had little suspicion of the extent to which his machine had been penetrated. He said that he had been alert ed by several elect ronic mail message from colleges elsewhere in the country. Each of the mail messages indicated that an attack had occurred over the weekend and the logs showed the origin to be his site in nort h London.The syst em manager explained t hat t hey had an et hernet local area net work (LAN) wit h VAXes, Unix machines, and IBM PCs. A SUN 3/470 workst at ion act ed as a link bet ween t he LAN and t he out side world. It was linked bydedicat ed dat a lines t o t wo X.25 wide area net works (WANs) - Brit ish Telecom’s PSS and t he academic communit y’s JANET. The workst at ion had been setup to only allow root logins from the console.The connections to the machineSecure The SystemWhen I received t he t elephone call my first advice was t o t reat t his incident like a laborat ory experiment and record det ails in a not ebook - discussions, evidence, and actions. I suggested that they should•secure as much evidence as possible - if pract ical a complet e dump of the filestore should be taken•check all their logs for spurious activity•search their filestore for suspicious files•keep discussions out of e-mail.I post ed him copies of some useful document s t hat would provide addit ional advice on dealing with computer intrusions and system security •David Curry’s paper on securing Unix•The introduction to COPS by Dan Farmer•Coping with security incidents by Russell Brand.Initial EvidenceIn subsequent telephone conversations that day the site told me what strange ent ries t hey had found. SUN Microsyst ems’ X.25 soft ware records net work act ivit y in several logs. Det ails of incoming int eract ive calls being writ t en t o x29serverlog, and outgoing to x29userlog. Their x29userlog showed numerous outgoing connections over the weekend and during the nights of the previous week. Their x29serverlog showed no corresponding incoming act ivit y. Thus the machine had recorded outgoing network activity but no incoming sessionsfrom the wide area network. The records of the security guards confirmed that no one had been in the building at these times.From this initial information it appeared that the computer system had been the subject of unauthorised use. Since 29th August 1990 unauthorised use or modificat ion of a comput er syst em has been a criminal offence in t he Unit ed Kingdom. When asked, t he syst em manager agreed t hat t his incident should be report ed t o t he police. I passed init ial det ails of t his incident t o t he Metropolitan Police Computer Crime Unit.What About Tonight?It was clear t hat t here were t rouble makers using t he machine, but where were they coming from?They could have been coming from t he LAN wit hin t he building or from t he WAN. The records of physical access t o t he building ruled out local sources. This left the wide area networks.The system manager was now faced with two problems. Firstly, the logs were inconsistent - so their machine had been misused. Secondly, the extensive log of out going act ivit y plus e-mail messages point ed t o many ‘downst ream’victims. That is, sites that had been attacked with his machine as the apparent origin.SUN’s X.25 software has a facility for tracing all network activity. If this trace could be left running overnight it would record det ails of any furt her int rusions. It would ident ify which net work t he at t acks were coming from. It would also provide some information about where the attackers were located, how they were getting in, and what they were doing when using this machine. Obviously a process called x25trace would be spot t ed by any int ruder. Therefore, t he t race was left running as an innocuous process t hat should always be there on this machine - screenblank. This process was chosen as it is normally t here t o blank t he screen of t he workst at ion’s console. Similarly, the trace output could not be written to a file with an obvious name. The output was directed to overwrite a dummy file in a user directory of 30 or 40 files of experimental data that all had similar names. The dummy file was created to have a date consistent with the other files - about six months earlier. Downstream ActivityAll sit es, as a first priorit y, must prot ect t heir own syst ems. Warning ‘downst ream’ sit es must t ake second place. A small sit e and first t ime vict im will not have the resources or experience to alert other sites. Therefore, I asked t hem t o send me a copy of t heir x29userlog and I would cont act t he ‘downstream’ sites. The log I received extended back to October 1989 when the machine was installed. A sample extract from this log is given in Appendix A.Looking through the entries revealed many interesting facts.•The log showed that the facility for outgoing interactive calls had been only sporadically used over t he previous eight een mont hs -but always during the day.•There was a suspicious gap in the log from the end of January to early April - the reason for this would be confirmed several weekslater.•The portion of the log for the past few days was 665 lines long. It started at 04:48 on the Wednesday morning and extended to 03:09t hat Monday morning.•The times of day when the intrusions occurred gave a clue to the country of origin.Correlating the entries showed that in this time•337 probes had been made•against 122 different machines•at 70 different sites.From the information in the log it was reasonable to assume that at least 68 of these machines (on 40 sites) had been successfully penetrated.TuesdayThe syst em manager report ed back - t he x25trace had been successful. Over Monday night t wo sessions by hackers had been capt ured. The first, on Monday evening at 21:25 came via the PSS line from an address (apparently) in Singapore. The second one at 02:23 came from a UK PSS address.The Singapore address had not been seen before and was traced back to a VAX syst em. The UK number had been t he source of many incident s over t he previous 18 mont hs. It belonged t o an account used for dial-in access t o a British Telecom PAD on PSS.This informat ion showed t hat t he hackers were coming from out side t he academic communit y. They were using t his machine as a gat eway bet ween the public network and JANET.Session transcriptsThe x25trace out put was laboriously analysed by t he syst em manager; it extended to 240 pages. The many electronic mail messages and the enveloping packet informat ion were discarded. This left a t ranscript of each command keyed by t he hacker and t he result ing response produced. It showed t hat on both occasions the hacker(s) had logged in directly as root, despite the system having been setup to prohibit this.Not cont ent wit h masquerading as root t he hacker(s) had used t heir own password. So now it was possible t o login as root using t wo different passwords. The hacker’s login password was allowed from any t erminal or from any other machine while the legitimate one continued to be only accepted from t he workst at ion console. The password file it self did not appear t o have been t ampered wit h and t here cont inued t o be just one ent ry for root. The hacker’s session was not listed when the command who was typed - somethingt hey did frequent ly t o check for t he presence of ot her users. They had, it seemed, short-circuited the accounting system to make themselves invisible. The discrepancy between the logs of the incoming and outgoing network calls could now be accounted for. While logged on, the hacker prepared a ‘cleaned-up’ logfile by editing a copy of the original. The hacker then scheduled a batch job t o run immediat ely following logging-off which overwrot e t he genuine x29serverlog with the edited copy, adjusted the file protections and back dated its creation date.The transcript also showed that there was a hidden file available to the hacker that held valid login names and passwords for this machine.The special password for root and the file with the list of passwords pointed to the hacker having replaced /bin/login with a Trojan horse. To do this either the file permissions were set wrongly or the hackers had previously found a way of becoming the super-user root.Apart from cleaning up the logs, noting user passwords, and checking for the presence of ot her users t he hacker was most ly concerned wit h finding out what was known about him and ensuring t hat he was not monit ored. He skimmed through all the mail boxes of the systems staff and operations staff, get t ing some point ers as t o who was who by t he use of finger. He came very close to locating the x25trace output while looking for newly created files.The sit e made cont act wit h a securit y expert from SUN Microsyst ems. While t hey cont inued t o liaise wit h SUN and t he Met ropolit an Police’s Comput er Crime Unit I set about contacting the ‘downstream’ victims.x29userlogIn one t ype of ent ry, x29userlog records t he st art t ime and t he dest inat ion address of each out going connect ion from t he syst em. A separat e ent ry is writ t en for t he end of t he connect ion and t he amount of dat a t ransferred in each direct ion.If a hacker probes a machine t hey will receive t he login banner, and t ry t o login. If t hey fail t o login t hey may t ry t wo or t hree more t imes before t he connection is broken. Should the hacker fail to login then the x29userlog will only show a small amount of data received. If the log shows a large amount of dat a received t hen t he hacker must have been successful in logging in t o t he target system.I chose an arbit rary limit of 1000 byt es t o different iat e successful and failed logins. Using t his limit, t he list of probes was priorit ized so t hat t he worst affected sites could be contacted first.I st art ed t he process of t elephoning sit es in t he academic communit y. The x29userlog provided firm evidence t hat a downst ream machine had been penetrated. Therefore, where I did not know a specific security contact I went t o t he t op and t elephoned t he direct or of each comput er cent re. Each t ime, I read out t he det ails from t he x29userlog; confirmed t he informat ion by e-mailing an extract of the relevant parts; warned them to look for a replaced /bin/login; and asked them to report back anything they found.Before I had got one fifth of the way through the list, telephone calls started coming in giving details of what the first sites had found. Armed with this new informat ion I went back t o t he beginning of t he list and gave t he addit ional details to each site.Many sit es were unable locat e any corresponding ent ries in t heir logs. So it was presumed that the intruders had managed to become super-user and edit the logs.A few sit es report ed t hat t hey found suspicious use of cert ain user-names in t heir logs. Some of t he usage came from origins ot her t han t his sit e t o t he nort h of London - a common source was t he main gat eway bet ween PSS and JANET.Becoming Super-UserOne site reported that their machine had crashed while the hacker was logged on. As a result the hacker had been unable to tidy up. They had left behind a small object program. The site reverse engineered the object code to yield a C source program. This program gave t he hacker (as an ordinary user) t he abilit y t o overwrit e any file in t he syst em. This program was t he key t he int ruders had used t o become super-user and ult imat ely replace /bin/login. Details of this program are given in Appendix B.It appeared that this small program exploited a generic weakness in all BSD (Berkeley Software Distribution) derived Unix systems. A copy of the program was sent to CERT/CC in Pittsburgh.1 They confirmed that it did allow someone to overwrite any file, and that it would work on most BSD systems. They agreed that they would report the problem to Unix suppliers in the United States. This should event ually result in changes t o t he syst ems supplied by most companies. However, with so many attacks targeted at the United Kingdom a quicker solution was needed.Fixing the ProblemFort unat ely t he sit e t hat obt ain t he hacker’s program also had a copy of t he source of BSD Unix for one of their Sun systems. The generic weakness wasn’t difficult to understand - there were no checks on the type and permissions of the file being used as output. They were quickly able to modify the source and check that the changes did indeed stop the hacker’s program.Armed with a program to exploit the weakness, and a set of changes to fix the problem I set about cont act ing Unix suppliers in t he UK. This produced a problem - locat e someone in each company who is willing and capable of underst anding t he problem. Fort unat ely various sit es around t he academic 1 CERT/CC (Comput er Emergency Response Team/Coordinat ion Cent er) operat es from t he Soft ware Engineering Inst it ut e (SEI) of Carnegie Mellon Universit y (CMU) in Pit t sburgh, Pennsylvania, USA. It is funded by the US Department of Defense’s (DoD) Defense Advanced Research Projects Agency (DARPA). Their aim is to provide support and coordination for the response to intrusions affecting computer systems connected to the Internet research network.communit y were able t o give me names of suit able people. Wit hin a week I faxed copies of the program and the fixes to the manufacturers.Simply t elling t he suppliers was not enough; machines in t he communit y were still being attacked. The manufacturers had to actually produce patches for their systems that would fix the problem.Telling the worldCont act was maint ained wit h t he Unix suppliers. The progress of t heir product ion of fixes was followed. Once most manufact urers had pat ches available it was t ime t o t ell t he syst em managers around t he academic communit y.A few syst em managers were cont act ed personally and t old of t he need t o obt ain and apply pat ches t o t heir syst ems. However, it was impract ical t o individually contact the thousands of system managers.No central electronic mail lists had been established for distributing this type of informat ion. I sent warnings t o ot her list s; for example, t he direct ors of comput er cent res. I found out lat er t hat oft en t his informat ion had not been passed on.With so many people becoming aware of the problem it was clear that the press would find out sooner or lat er. Therefore, a press release was prepared. A similar st at ement was prepared by t he Comput er Crime Unit and passed t o t he Met ropolit an Police’s press office. This was picked up by a report er from The Times. His article triggered interest from the computer trade press. They published several art icles warning syst em managers t o obt ain t he lat est securit y pat ches.The newspaper report s result ed in calls from several furt her comput er manufact urers; including one from Aust ralia.Other ThreadsBig BillsAt the request of the police, British Telecom produced an advance copy of the bill for t his medical research sit e’s line t o PSS. Rat her t han just a simple quart erly t ot al, BT produced an it emised bill. This list ed every call made on that line, showing the date and time, number called, duration, and volume of data transferred in each direction. The listing was nearly 12 inch thick.The det ails on t he bill showed why someone had delet ed a sect ion of t he x29userlog. The listing showed that someone had been making calls one after anot her, every t wo seconds for several hours each night. The numbers called were consecut ive and pot ent ially belonged t o organisat ions in t he UK. A hacker had obviously been running an aut omat ed process on t his machine t o t ry all possible numbers and search t he dat a net works for ot her comput ers.2 2 A similar process, but using the telephone network, was shown in the 1983 film War Games.Wit h dat a net works operat ed as ‘ex-direct ory’, such brut e force searches are necessary to locate new computers for hackers to attack.PSS Gateway LogsOnce this medical research site had looked through the transcripts of the two Monday night sessions they decided to disconnect their system from both PSS and JANET. This would isolate their LAN from the rest of the world and keep the systems and data safe. Of course, this would also sever legitimate external cont act, including elect ronic mail.Having cut t his rout e from t he out side world int o JANET t he hackers were forced to use a different route to continue their intrusions. Fortunately for the invest igat ion t hey swit ched t o using t he main gat eway3 bet ween PSS and JANET. This gat eway maint ains logs t hat give origin and dest inat ion, st art and finish time, and traffic volume for each call. These logs are often searched for suspicious patterns of activity - in both directions.The Foreign ConnectionThe x25trace had already shown that the attacks appeared to be coming from two origins outside the academic community. Checking the PSS gateway logs revealed that attacks were coming in from a number of other sites around the world. This check was done in several ways. First ly, t he recent logs were searched for occurrences of both the UK and Singapore addresses. This yielded a list of addit ional academic machines t hat had been at t acked by hackers. Secondly, t he logs were searched for act ivit y t o (and from) t he academic machines that were already known to have been penetrated. This produced a list of addit ional sources for t he at t acks. Finally, t he logs were searched manually for suspicious act ivit y. The whole sequence was repeat ed several t imes.In t he end, t he whole process of searching t he logs yielded a list of t en comput er syst ems around t he world. These machines were being used as st epping-st ones when at t acking t he UK academic communit y’s comput ers. These at t acks were not isolat ed incident s; t here were hundreds of at t acks spread over several weeks. I thought it was unlikely that the companies that owned t hese machines knew t hat t heir comput ers had been infilt rat ed. I passed the details of these systems to the Computer Crime Unit. They in turn, relayed the information via Interpol to their foreign counterparts. In all cases it came as a surprise t o t he companies when t he police cont act ed t hem and informed t hem t hat someone was misusing t heir comput er syst em. In some cases, this was the first true computer crime that these police teams had had to deal with.The End?Once t he foreign sit es had been cont act ed; once t he art icles had appeared in t he newspapers, t he at t acks st opped happening. Wit h t he passwords on t he 3To t he users t here appears t o be one gat eway. However, in fact, t here are several ident ical gateways at different sites to provide load sharing and redundancy.machines t hey were using changed and t he press publicit y t he hackers must have known that their activities had been discovered.Was t his t he end of t he incident? We - t he vict ims, t he police, t he manufact urers, and so on - may never know for cert ain who was responsible for the intrusions over this weekend. However, in June 1991 three people were arrest ed in simult aneous raids on t heir homes England. On 22nd February 1993 these three young men appeared in the dock at Southwark Crown Court, London. They were joint ly charged wit h conspiracy t o commit offences under t he Comput er Misuse Act 1990 and t he Telecommunicat ions Act 1984. Neil Matthew Woods, 24, of Oldham, Manchester, and Karl Austin Strickland, 22, of Liverpool pleaded guilty. On Friday 21st May 1993 they were sentenced to six-months imprisonment. Earlier, on Wednesday 17th March 1993, after a three-week trial, Paul Daniel Bedworth, 19, of Ilkley, West Yorkshire, was acquitted of all charges. All t hree admit t ed t he act ions described by t he prosecut ion -hacking int o comput er syst ems, modifying (syst em) dat a and programs, and running up huge t elecommunicat ions bills for various organisat ions.When questioned by the police, Paul Bedworth denied hacking into this system in North London. Both Karl Strickland and Neil Woods admitted that this was one of the systems they had hacked into.LessonsAlthough the incident described here occured more than two years ago similar event s could st ill happen t oday. Several lessons can be learnt from t hese intrusion into computer systems.•Only 5% of int rusions int o comput er syst ems are discovered. In t his instance, before the sites were warned, the attacks were noticed by the syst em managers of only 5 out of t he 122 machines probed. Similar figures are revealed by ot her int rusions - in t he UK and abroad - on academic sit es and commercial alike.•Not all intrusions leave records in log files. By modifying system code or simply edit ing t he log file t hese hackers hid t heir act ivit y from t he syst em manager.•Revealing details of an intrusion can help improve security. If the site in nort h London had kept t he incident secret t his part icular weakness in Unix would have remained available for hackers to exploit. BibliographyRussell L. Brand, Coping with the Threat of Computer Security Incidents - A Primer from Prevention through Recovery, June 1990, (available online from in pub/info/primer as primer.ps and primer.txt).David Curry, Improving the Security of your UNIX System, SRI International, Report ITSTD-721-FR-90-21, May 1990, (available online from in pub/info as security-doc.tar.Z and security-doc.txt).Dan Farmer, COPS and Robbers - UN*X System Security, (available with the COPS package), 1989-1991, (available online from in pub/cops).Clifford St oll, The Cuckoo's Egg - Tracking a spy through the maze of computer espionage, Doubleday, 1989, 326pp, ISBN 0-370-31433-6.Bryan Clough and Paul Mungo, Approaching Zero, Faber and Faber Ltd, 1992, ISBN 0-571-16546-X, £14.00.Kat ie Hafner and John Markoff, Cyberpunk, Fourt h Est at e Lt d, 1991, ISBNּ1-872180-94-9, £14.99.Lory R Rice & Simon Tye, Diary of a Security Alert, Proceedings of SUN UK User Conference 1991.Appendix ASample extracts from the x29userlog.x29userlog - 1991Jan 22 11:07:32 Pid 515: Call connected to 000001009302, uid=14Jan 22 11:08:36 Pid 515: Call cleared - segs: sent=30 rcvd=121, chars: sent=72 rcvd=3861Apr 10 04:48:53 Pid 10753: Call connected to 000015000037, uid=10Apr 10 04:54:36 Pid 10753: Call cleared - segs: sent=129 rcvd=620, chars: sent=328 rcvd=10086 Apr 10 05:02:13 Pid 10765: Call connected to 000015000037, uid=0Apr 10 05:04:05 Pid 10765: Call cleared - segs: sent=45 rcvd=205, chars: sent=122 rcvd=2309 ....Apr 10 07:57:46 Pid 10951: Call connected to 000010401000, uid=0Apr 10 08:00:08 Pid 10951: Call cleared - segs: sent=9 rcvd=32, chars: sent=61 rcvd=541Apr 11 01:13:33 Pid 2872: Call connected to 000010300050, uid=0Apr 11 01:14:22 Pid 2872: Call cleared - segs: sent=5 rcvd=19, chars: sent=26 rcvd=313Apr 11 01:16:25 Pid 2879: Call connected to 000010300050, uid=0Apr 11 01:17:11 Pid 2879: Call cleared - segs: sent=9 rcvd=31, chars: sent=59 rcvd=483....Apr 11 07:55:45 Pid 3165: Call connected to 000041001300, uid=0Apr 11 08:06:01 Pid 3165: Call cleared - segs: sent=95 rcvd=706, chars: sent=1997 rcvd=34254Apr 11 08:08:47 Pid 3172: Call connected to 000045701000, uid=0Apr 11 08:08:58 Pid 3172: Call cleared - segs: sent=0 rcvd=4, chars: sent=0 rcvd=27Apr 11 19:01:55 Pid 5026: Call connected to 000021000018, uid=0Apr 11 19:02:25 Pid 5026: Call cleared - segs: sent=5 rcvd=15, chars: sent=25 rcvd=119....Apr 12 08:23:50 Pid 5791: Call connected to 000060210009, uid=0Apr 12 08:23:58 Pid 5791: Call cleared - segs: sent=0 rcvd=6, chars: sent=0 rcvd=93Apr 12 08:24:47 Pid 5794: Call connected to 000060210019, uid=0Apr 12 08:58:49 Pid 5794: Call cleared - segs: sent=315 rcvd=1902, chars: sent=4821 rcvd=84764 Apr 12 18:07:41 Pid 7132: Call connected to 000007006460, uid=0Apr 12 18:09:21 Pid 7132: Call cleared - segs: sent=8 rcvd=68, chars: sent=81 rcvd=2647Apr 12 20:39:01 Pid 7225: Call connected to 000005261000, uid=0Apr 12 20:47:53 Pid 7225: Call cleared - segs: sent=93 rcvd=784, chars: sent=326 rcvd=33909 ....Apr 13 07:29:31 Pid 7901: Call connected to 000013000050, uid=0Apr 13 07:29:31 Pid 7901: Call cleared - segs: sent=0 rcvd=2, chars: sent=0 rcvd=27Apr 13 07:32:30 Pid 7915: Call connected to 000060303201, uid=0Apr 13 08:38:16 Pid 7915: Call cleared - segs: sent=2427 rcvd=5390, chars: sent=9250 rcvd=119838 Apr 13 13:22:21 Pid 8051: Call connected to 000050005002, uid=0Apr 13 13:23:18 Pid 8051: Call cleared - segs: sent=5 rcvd=59, chars: sent=36 rcvd=2670....Apr 14 06:09:39 Pid 9021: Call connected to 000005114000, uid=0Apr 14 06:15:11 Pid 9021: Call cleared - segs: sent=42 rcvd=926, chars: sent=458 rcvd=9699Apr 14 06:27:37 Pid 9034: Call connected to 000050005002, uid=0Apr 14 13:09:01 Pid 9158: Call connected to 000005114000, uid=0Apr 14 13:26:48 Pid 9158: Call cleared - segs: sent=277 rcvd=1522, chars: sent=5840 rcvd=24468 ....Apr 14 13:32:43 Pid 9181: Call connected to 000052100200, uid=0Apr 14 13:32:51 Pid 9181: Call cleared - segs: sent=0 rcvd=6, chars: sent=0 rcvd=84Apr 14 18:40:34 Pid 9278: Call connected to 000005393000, uid=0Apr 14 18:47:10 Pid 9278: Call cleared - segs: sent=19 rcvd=671, chars: sent=152 rcvd=34544Apr 14 19:44:27 Pid 9313: Call connected to 000005393000, uid=0Apr 14 20:15:41 Pid 9313: Call cleared - segs: sent=224 rcvd=787, chars: sent=4001 rcvd=19276 ....Apr 14 20:19:51 Pid 9335: Call connected to 000012200086, uid=0Apr 14 20:20:33 Pid 9335: Call cleared - segs: sent=7 rcvd=35, chars: sent=39 rcvd=328Apr 14 23:44:03 Pid 9411: Call connected to 000013104000, uid=0Apr 14 23:44:16 Pid 9411: Call cleared - segs: sent=1 rcvd=8, chars: sent=5 rcvd=140....Apr 15 03:07:55 Pid 9805: Call connected to 000050005002, uid=0Apr 15 03:09:24 Pid 9805: Call cleared - segs: sent=13 rcvd=180, chars: sent=54 rcvd=8789。
