[gridengine users] Node with negative value of a consumable

William Hay w.hay at ucl.ac.uk
Tue Jun 14 13:09:14 UTC 2011

On 14 June 2011 13:14, Javier Lopez Cacheiro <jlopez at cesga.es> wrote:
> Hi William,
>>>> Assuming this is for local scratch space on Linux one thing I've been
>>>> considering is using almost the whole disk as swap and then mounting a
>>>> suitably sized tmpfs on $TMPDIR in the prolog
>>>> followed by a umount in the epilog.  Have to be careful to ensure it
>>>> didn't cause real memory to be devoted to the VFS rather than
>>>> processes though.
>>> Taking your approach has made me think of an alternative where LVM could
>>> be used in an easy way, creating a VG for scratch and configuring the
>>> LVs on demand for each job using the size requested. At the end the LV
>>> would be easily removed.
>>> To reduce the delay during filesystem creation, probably xfs could be
>>> used that is much faster to create.
>>> It is interesting to explore these options using prolog/epilog, even if
>>> the final solution for all purposes would probably be implemented inside
>>> shepperd.
>> Inside  shepherd? uggh.  One of the things I like about GE is that it
>> follows the software tools philosophy pretty closely.
>> It does relatively little but has hooks in almost all the right places
>> to let you add needed functionality.  I suppose
>> one advantage of shepherd or starter_method is that you could combine
>> it with Linux namespaces  to make a very private
>> directory.  When combined with tmpfs it might even auto free the space
>> without an explicit umount.
> I think about shepperd because all other system limits are also
> implemented there. The main problem is that setrlimit does not implement
> a proper resource to control disk usage, FSIZE is not very useful and GE
> just reflects that limit.
The main problem I have here is that most of the other limits are
fairly portable across unices at least in how they are implemented by
shepherd if not always their semantics.

Volume Managers vary a lot filesystems (including tmpfs) vary as well
but not as much.  I'm not familiar at all with non-linux namespace
like functionality but I'd be willing to bet there is a fair amount of
variation there.   Shepherd would presumably need to ifdef all these
alternatives.  To create a logical volume and XFS filesystem you'll
presumably need to run external commands unless you plan to embed code
for all that in the shepherd.  Once you've gone to the  bother of
running external commands you might as well make them configurable to
improve portability/flexibility in which case you are back to
something that looks an awful lot like prolog or starter_method.

> With a additional setrlimit to control total disk usage that could be
> pretty straight forward.
> The idea of isolating the process using Linux namespaces could be also a
> plus in some circumstances especially when data privacy is important for
> users.

More information about the users mailing list