We have a system that stages some data.  A job fires off and picks up the data from the stage db, pushes it to the real db, performs a few edits on it, and then goes onto the next record.  Sounds simple.

Customer calls to complain that the system is too slow.  10s of hours waiting for items to reach them...

I start digging to get better details of the system.  The above is a generalization.
What happens is the job calls a proc.  This proc says, get me the top 1 item from stage db.  Then calls another proc with this ID.  This proc in turn does some insertion into multiple tables.  The last of which has a trigger.  The trigger sends this ID thru a series of edits.  Once done, there is some clean up procs called for this ID and eventually the stage data is removed.  Thus returning to the original proc.  It then says, are there any records left in the stage table?  if so, get me the top 1 item from the stage table.  And repeats.
The trigger turns out to process some 139 rules.  53 of which are rules comparing one field to be equal to a value.  the rest are comparing one field to be like a value.  There is a priority to these rules.  If you look at the history, the last rule, is the one most picked, eventually, by the system.  So, 139 rules must be traversed before it can perform an action.  139 checks on values.  And keep in mind that this is in a trigger, while we are moving data from stage to real db.  If it takes 2 minutes to process all this, and we have received 50 new items in those 2 minutes, you can imagine the backlog. 

Thus the delay.  Poor job, on occasion, was running for 9 hours straight.  Trying to get everything thru the system.
So, a little research reveals that the priority of the edits could be drastically altered, and the most picked edits moved up the priority list.  This requires some work on the operations staff.  They are duely informed.  Nothing happens. 

Last night, same process occurred.  Same slowness.  I'm just waiting for the email to complain about slowness....