multithreading - Deadlock Delphi explanation/solution -

on server application have following: class called jobmanager singleton. class, scheduler, keeps checking if time add sort of job jobmanager.

when time so, scheduler like:

tjobmanager.singleton.newjobitem(parameterlist goes here...); 

at same time, on client application, user generates call server. internally, server sends message itself, , 1 of classes listening message jobmanager. jobmanager handles message, , knows time add new job list, calling own method:

newjobitem(parameter list...); 

on newjobitem method, have this:

  cs.acquire;   try     dosomething;     callamethodwithanothercriticalsessioninternally;       cs.release;   end; 

it happens system reaches deadlock @ point (cs.acquire). communication between client , server application, made via indy 10. think, rpc call fire server application method sends message jobmanager running on context of indy thread.

the scheduler has own thread running, , makes direct call jobmanager method. situation prone deadlocks? can me understand why deadlock happening here?

we knew that, sometimes, when client did specific action, cause system lock, find out point, critical section on same class reached twice, different points (the scheduler , message handler method of jobmanager).

some more info

i want add (this may silly, anyway...) inside dosomething there

  cs.acquire;   try     other stuff...       cs.release;    end; 

this internal cs.release doing external cs.acquire? if so, point scheduler entering critical section, , lock , unlock becomes mess.

there isn't enough information system able tell definitively if jobmanager , scheduler causing deadlock, if both calling same newjobitem method, should not problem since both acquire locks in same order.

for question if newjobitem cs.acquire , dosomething cs.acquire interact each other: depends. if lock object used in both methods different, no 2 calls should independant. if it's same object depends on type of lock. if locks re-entrant locks (eg. allow acquire called multiple times same thread , count how many time have been acquired , released) should not problem. on other hand if have simple lock objects don't support re-entry, dosomething cs.release release lock thread , callamethodwithanothercriticalsessioninternally running without protection of cs lock acquired in newjobitem.

deadlocks occur when there 2 or more threads running , each thread waiting thread finish it's current job before can continue self.

for example:

thread 1 executes:  lock_a.acquire() lock_b.acquire() lock_b.release() lock_a.release()   thread 2 executes:  lock_b.acquire() lock_a.acquire() lock_a.release() lock_b.release() 

notice locks in thread 2 acquired in opposite order thread 1. if thread 1 acquires lock_a , interrupted , thread 2 runs , acquires lock_b , starts waiting lock_a available before can continue. thread 1 continues running , next thing try acquire lock_b, taken thread 2 , waits. in situation in thread 1 waiting thread 2 release lock_b , thread 2 waiting thread 1 release lock_a.

this deadlock.

there several common solutions:

1. only use 1 shared global lock in code. way impossible have 2 threads waiting 2 locks. makes code wait lot lock available.
2. only ever allow code hold 1 lock @ time. hard control since might not know or control behavior of method calls.
3. only allow code acquire multiple locks @ same time, , release them @ same time, , disallow acquiring new locks while have locks acquired.
4. make sure locks acquired in same global order. more common technique.

with solution 4. need careful programming , make sure acquire locks/critical sections in same order. debugging can place global order on locks in system (eg. unique integer each lock) , throwing error if try acquire lock has lower ranking lock current thread has acquired (eg. if new_lock.id < lock_already_acquired.id throw exception)

if can't put in global debugging aid find locks have been acquired out of order, i'd suggest find places in code acquire lock , print debugging message current time, method calling acquire/release, thread id, , lock id being acquired. same thing release calls. run system until deadlock , find in log file locks have been acquired threads , in order. decide thread accessing it's locks in wrong order , change it.