Track your comments!
[x]


When you register, comments on your articles and replies to your comments appear here. Register Now!

Sign in to your account
[x]

Not a Scientific Blogging member yet?

Register Now for a Free Scientificblogging.com Account

  • Customize your profile with pictures, banner, a blogroll and more.
  • Leave comments on articles, add other members to your friend lists, chat with people on the site.
  • Write blog posts that can be seen by hundreds of thousands of readers.

It's free and it only takes a minute!

Already a Scientific Blogging member?

Sign In Now

Banner
By Tommaso Dorigo | September 22nd 2009 06:44 PM | 14 comments | Print | E-mail | Track Comments
.

More A Quantum Diaries Survivor articles

All

About Tommaso Dorigo

I am an experimental particle physicist working with the CMS experiment at CERN and the CDF experiment at Fermilab. In my spare time I play chess, abuse the piano, and aim my dobson telescope at... Full Bio

Most of us like to be in control: of what happens around us, of our own feelings, of our actions, of the actions and well-being of our beloved ones. Being in control means feeling secure, unthreatened. It is the prevalence of order on chaos. And chaos, I have grown to realize, is one of the things that scares me most. Yes, I am a true control freak.

This week I am spending my evenings in the CDF control room, as a Scientific Coordinator. That literally means being in control of what happens to the detector and its subsystems, and ensuring that data is collected with the maximum efficiency. Of particular concern are situations when there are warnings or other alarms from a subsystem which is deemed "critical" for the flagging of data as "good for physics". In such circumstances, we need to take prompt action, trying to figure what the problem is, and if needed call the relevant expert. There are at least three dozen page carriers in the area around Fermilab, who are instructed to run to the aid of their subsystem, in case something odd happens. And the CDF electronic logbook is read daily by a hundred more of our collaborators, who care for their detector and the data we collect.

The shift crew is composed of a Scientific Coordinator (SciCo), plus an Ace -the person who actually executes all computer commands to control the systems and the data taking- and a Consumer Operator -the person who controls the quality of the data being collected, by comparing running histograms with reference ones. The SciCo mostly sits and watches, in the middle of the big control room, and supervises the proceedings. He or she is the one who sometimes has to take decisions on how to react to emergencies. Among the duties there is the supervision of data taking, the responsibility for enforcing the emergency response procedures (say in case of fire, or oxygen deficiency alarms), the care of voltages and currents of our silicon detector, the communications with the main control room and with experts, and a very particular procedure to perform only when the silicon, our most precious subsystem, is in danger of being burnt by a malfunction in its cooling system.

I have not visited too many control rooms of complex experiments or systems in my life, but I bet that most of them are instrumented with something resembling the thing seen below, a "big red button": something to push when s**t happens.


Gary Larson even made a cartoon about it (you can see it on the left), and as always his humour is spot-on: in every control room, there is somebody who is constantly thinking at the chance that he or she will one day have to hit the big button! The legend says "One day, Irwin knew, he was just going to have to push that big button". Consequences are dire: once you press the button, you will be called to explain why you did. And if you do not push it in case it is needed, responsibilities are even heavier.

Today things are running smoothly in the control room. But yesterday, if we speak of loss of control... Well: at a certain point, at least four subsystems decided to give warnings or alarms: the silicon, the XFT, the Level 3, the data quality monitoring system. It must have been funny watching us frantically going from one terminal screen to another, issuing computer commands, calling people over the phone, answering other calls and keeping people on hold. Fortunately, these activities are short-lived, else I would not stand the stress.

So here I am pictured after the smoke cleared, toward the end of my shift yesterday: as you see, I do not look stressed at all. Bored, maybe.



Comments

Hi Tommaso,
the big button looks really tempting. What exactly would it do? I imagine a large tank of liquid nitrogen or something to open, shock-frosting the silicon. I believe Big Reg Buttons should always be connected to some impressive emergency system.
BTW, on the picture you look a bit like "I only I had brought my mouse, so I log into ICC now..." :)

dorigo's picture
The button powers off the silicon, in case other systems do not do it automatically in case of a problem with the cooling. Powering off that device is not something one would do light-heartedly in normal circumstances, but if the systems are not responding it may be the only way to go.

Yes, I ache for not bringing my mouse. You read my thoughts! Anyway I am  not even sure I could log in to the ICC, since Fermilab blocks access to many sites.

Cheers,
T.

The "big red button" (which I have actually pushed once) removes power from the racks which power the power supplies which power the silicon detector. The power supplies themselves are interlocked so that they drop power when a problem occurs (e.g. temperature has risen too high). Since the inability to remove the power in such an instance could be catastrophic for the delicate silicon detector, the big red button is there as a back up to this normal interlock system. If the interlock system fails for some reason (e.g. the computer that the interlock instructs to power down the silicon itself fails) *and* there was a problem simultaneously in another system (e.g. a failure in cooling) the power must be removed by brute force at the rack level via the red button since this is the only way to protect the system. Needless to say, this kind of coincidence of failures rarely occurs (the only occurrence I know of in the last 8 years is the one where I pressed the button though I suspect it has happened a few other times), but in HEP one must be prepared for everything, especially when it comes to silicon detectors!

lumidek's picture
Is "control freak" the same thing as "il duce" in Italian?

Pushing the big red button has been my biggest dream since I met HEP...

dorigo's picture
Hi Lubos, no, in Italian "duce" means "leader", not "paranoid" :)
Cheers,
T.

Nicola, I am told somebody pushed it last week. Damn! I missed it!
T.

In the US at least, it's an Occupation, Safety & Health Administration (OSHA) requirement that all big hulking pieces of electrical machinery have a big red button available to turn it off manually. It has to be properly located, etc.

This is one of the sad facts of life you learn if you end up working in the real world assembling big things.

Well, CMS has a "Black Hole/Strangelet Crash button". I don't know why we put it in the cavern instead of the control room, though - probably to make for a more interesting movie adaptation when the heroic scientists get trapped in the elevator.

dorigo's picture
:) Thomas, I was unaware of this important feature of our detector. I believe the movie must imply that the scientist has a means of traveling backwards in time through the BH just a few minutes before he enters the hall, so he can push it.
Cheers,
T.

Hi Tommaso,
well, you forgot to mention one of the most important duties of the SciCo: bring in delicious food for the whole shift crew! And on the photo it looks like your food was really tasty, it seems everything is already eaten up.

cheers,
Jan

dorigo's picture
... Or never been there!

Well I do my best, but the problem is that if I bring too good stuff I end up munching the whole duration of the shift, which is BAD for my diet! So I end up mostly bringing lots of fruit, plus bread and sliced stuff for sandwiches. I really wish we could bring in also a beer or two once in a while, damn it. But I realize it would be really bad PR.

Cheers,
T.

Hank's picture
 I really wish we could bring in also a beer or two once in a while, damn it. But I realize it would be really bad PR.

It would do wonders for recruitment.

And think of the practical jokes!  After a few beers, you know someone would get some black paper and cut a big circle and tape it on the wall and then stand in front of it dressed as a Mayan or something equally world-destructive.

Otto Rössler and crew would go apoplectic with glee!

Eating on the job? That's not possible: it would get in the way of your ability to play speed chess.

dorigo's picture
I don't do that on duty Carl, and to make sure I don't, I leave home my mouse...
T.

Add a comment

The content of this field is kept private and will not be shown publicly.
  • Allowed HTML tags: <sup> <sub> <a> <em> <strong> <center> <cite> <code> <TH><ul> <ol> <li> <dl> <dt> <dd> <img> <br> <p> <blockquote> <strike> <object> <param> <embed> <del> <pre> <b> <i> <table> <tbody> <div> <tr> <td> <h1> <h2> <h3> <h4> <h5> <h6> <hr> <iframe>
  • Lines and paragraphs break automatically.
  • Web page addresses and e-mail addresses turn into links automatically.
CAPTCHA
If you register, you will never be bothered to prove you are human again. And you get a real editor toolbar to use instead of this HTML thing that wards off spam bots.