[gridengine users] jobs getting killed (failed assumedly after job because: job 311263.1 died through signal KILL (9))

[Peskin, Eric]

Reuti,

No, there is nothing in /var/log/messages about the oom-killer (or anything getting killed).  There is stuff about DHCPDISCOVER not finding leases.  That is strange, because we are using static IP addresses, so I am not sure why anything is looking for DHCP.  There are also complaints that the various compute nodes cannot find a suitable server for ntpd.  Both of these are chronic messages that we always have.  They do not seem to be limited to the times when jobs fail.  There are messages about directories being nfs mounted and unmounted.  Finally, the mail log shows the message about the job failure being sent to the user.

As for the messages file on the exec host.  For the job mentioned in my original email, no I do not see anything.  We do have another similar failure (more recent), where I do see some messages at the time the job failed.  However, they refer to different job numbers.  In this, more recent case, the message sent to the end user was:
==============================================
Job 332115 (qmake) Aborted
Exit Status      = 137
Signal           = KILL
User             = tangz01
Queue            = casava.q at compute-2-13.local
Host             = compute-2-13.local
Start Time       = 10/13/2011 10:02:04
End Time         = 10/13/2011 10:09:34
CPU              = 00:00:09
Max vmem         = 1.322G
failed assumedly after job because:
job 332115.1 died through signal KILL (9)
==============================================

At that time on compute-2-13 itself, the file $SGE_ROOT/$SGE_CELL/spool/compute-2-13/messages has the following:

10/13/2011 10:09:37|  main|compute-2-13|W|reaping job "332836" ptf complains: Job does not exist
10/13/2011 10:09:37|  main|compute-2-13|E|can't open file active_jobs/332836.1/error: No such file or directory
10/13/2011 10:09:37|  main|compute-2-13|W|reaping job "332842" ptf complains: Job does not exist
10/13/2011 10:09:37|  main|compute-2-13|E|can't open file active_jobs/332842.1/error: No such file or directory

These messages are actually three seconds after the failure, and they refer to different job numbers.  But I list them because they are so close in time.

Thanks,
	Eric


On Oct 7, 2011, at 10:40 AM, Reuti wrote:

> Hi,
> 
> Am 04.10.2011 um 00:47 schrieb Peskin, Eric:
> 
>>> Is the hard run time limit (h_rt) getting reached some times but  
>>> not others?
>> 
>> No, we do not have any limits set:
>> 
>> [root at fen1 ~]# qconf -sq `qconf -sql` | grep [hs]_|sort -u
>> h_core                INFINITY
>> h_cpu                 INFINITY
>> h_data                INFINITY
>> h_fsize               INFINITY
>> h_rss                 INFINITY
>> h_rt                  INFINITY
>> h_stack               INFINITY
>> h_vmem                INFINITY
>> s_core                INFINITY
>> s_cpu                 INFINITY
>> s_data                INFINITY
>> s_fsize               INFINITY
>> s_rss                 INFINITY
>> s_rt                  INFINITY
>> s_stack               INFINITY
>> s_vmem                INFINITY
>> [root at fen1 ~]#
> 
> is there anything in /var/log/messages about the oom-killer? Or the  
> SGE messages files on the exechost's spool directory?
> 
> -- Reuti
> 
> 
>> 
>> On Oct 3, 2011, at 1:41 PM, Mike Hanby wrote:
>> 
>>> Is the hard run time limit (h_rt) getting reached some times but  
>>> not others?
>>> 
>>>> -----Original Message-----
>>>> From: users-bounces at gridengine.org [mailto:users-
>>>> bounces at gridengine.org] On Behalf Of Peskin, Eric
>>>> Sent: Monday, October 03, 2011 11:14 AM
>>>> To: users at gridengine.org
>>>> Subject: [gridengine users] jobs getting killed (failed assumedly  
>>>> after
>>>> job because: job 311263.1 died through signal KILL (9))
>>>> 
>>>> All,
>>>> 
>>>> I have a user running qmake jobs.  Intermittently, the job fails and
>>>> SGE says it was killed with signal 9.  The user did not kill it.  We
>>>> (the sysadmins) did not kill it.  How can I figure out what is going
>>>> on?  The worst part is that this problem is intermittent.  Exactly  
>>>> the
>>>> same command works sometimes but fails sometimes.  I have appended  
>>>> the
>>>> message from SGE below.  Any suggestions would be greatly  
>>>> appreciated.
>>>> 
>>>> Thanks,
>>>> 	Eric Peskin
>>>> 
>>>> From: root [root at local]
>>>> Sent: Saturday, September 24, 2011 9:04 PM
>>>> To: Tang, Zuojian
>>>> Subject: Job 311263 (qmake) Aborted
>>>> 
>>>> Job 311263 (qmake) Aborted
>>>> Exit Status      = 137
>>>> Signal           = KILL
>>>> User             = tangz01
>>>> Queue            = regular.q at compute-0-13.local
>>>> Host             = compute-0-13.local
>>>> Start Time       = 09/24/2011 19:03:31
>>>> End Time         = 09/24/2011 21:04:10
>>>> CPU              = 00:00:29
>>>> Max vmem         = 2.579G
>>>> failed assumedly after job because:
>>>> job 311263.1 died through signal KILL (9)
>>>> 
>>>> 
>>>> ------------------------------------------------------------
>>>> This email message, including any attachments, is for the sole use  
>>>> of
>>>> the intended recipient(s) and may contain information that is
>>>> proprietary, confidential, and exempt from disclosure under  
>>>> applicable
>>>> law. Any unauthorized review, use, disclosure, or distribution is
>>>> prohibited. If you have received this email in error please notify  
>>>> the
>>>> sender by return email and delete the original message. Please note,
>>>> the recipient should check this email and any attachments for the
>>>> presence of viruses. The organization accepts no liability for any
>>>> damage caused by any virus transmitted by this email.
>>>> =================================
>>>> 
>>>> 
>>>> _______________________________________________
>>>> users mailing list
>>>> users at gridengine.org
>>>> https://gridengine.org/mailman/listinfo/users
>> 
>> 
>> ------------------------------------------------------------
>> This email message, including any attachments, is for the sole use  
>> of the intended recipient(s) and may contain information that is  
>> proprietary, confidential, and exempt from disclosure under  
>> applicable law. Any unauthorized review, use, disclosure, or  
>> distribution is prohibited. If you have received this email in error  
>> please notify the sender by return email and delete the original  
>> message. Please note, the recipient should check this email and any  
>> attachments for the presence of viruses. The organization accepts no  
>> liability for any damage caused by any virus transmitted by this  
>> email.
>> =================================
>> 
>> 
>> _______________________________________________
>> users mailing list
>> users at gridengine.org
>> https://gridengine.org/mailman/listinfo/users
> 


------------------------------------------------------------
This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain information that is proprietary, confidential, and exempt from disclosure under applicable law. Any unauthorized review, use, disclosure, or distribution is prohibited. If you have received this email in error please notify the sender by return email and delete the original message. Please note, the recipient should check this email and any attachments for the presence of viruses. The organization accepts no liability for any damage caused by any virus transmitted by this email.
=================================