Big Red usage policies

On this page:

• Accounts
• Home directories
• CPU limits and batch jobs
• Computational resources
• Mail usage
• Scheduled down time
• Questions and comments

Accounts

Access to Big Red is provided to all Indiana University faculty, graduate students, and staff. Undergraduates must have an IU faculty or staff sponsor. Instructional use is limited to courses that have been approved by the Director for Research Technologies.

Home directories

Home directories reside on the RAC storage system and have a default quota of 10GB. If you need additional permanent storage space, apply for an account on IU's Massive Data Storage Service (MDSS).

CPU limits and batch jobs

User processes on the login nodes are limited to 20 minutes of CPU time. Processes exceeding this limit are automatically terminated with no warning. If you require more than 20 minutes of CPU time, submit a batch job to the LoadLeveler batch queuing system with the llsubmit command. You may run interactive jobs requiring more than 20 minutes of CPU time on nodes b509, b510, b511, and b512.

Computational resources

Effective October 2, 2007 the batch queues have been renamed on Big Red and Quarry to align with TeraGrid partners. This should make it easier for users to transition between TeraGrid resources.

On Big Red:

• BIG is now known as NORMAL.
• MED is now known as LONG.
• FAST is now known as DEBUG.

Wall time limits and node limits per queue will not change. Aliases for the old names are in place for job submissions, but when you use commands like llclass and showq -w class=XXXXX, you must use the new names.

Please send any comments or suggestions to  researchtechnologies@iu.edu .

Each queue has properties defined in order to maximize job throughput and minimize time spent waiting in queue. Submit jobs to the queue most appropriate to the specific job requirements. If no queue is specified, jobs will go into the DEBUG queue.

Note: The total number of idle jobs you may have over all queues is 128. In the DEBUG queue, you may have only 1 idle job.

SERIAL queue properties

• Nodes: 760
• Maximum nodes per job: 1 (4 cores)
• Maximum nodes per user: 64
• Maximum wall time: 48 hours (2 days)
• Maximum CPU time: 192 hours (48 hours * 4 cores)
• Maximum running jobs in queue: Unlimited
• Maximum idle jobs in queue: Up to 128 (see "Note" above)

NORMAL queue properties

• Nodes: 392
• Maximum nodes per job: 256 (1,024 cores)
• Maximum nodes per user: 392
• Maximum wall time: 48 hours (2 days)
• Maximum CPU time: 48 hours (2 days) per core per job
• Maximum running jobs in queue: Unlimited
• Maximum idle jobs in queue: Up to 128 (see "Note" above)

LONG queue properties

• Nodes: 368
• Maximum nodes per job: 64 (256 cores)
• Maximum nodes per user: 64
• Maximum wall time: 336 hours (14 days)
• Maximum CPU time: 336 hours (14 days) per core per job
• Maximum running jobs in queue: Unlimited
• Maximum idle jobs in queue: Up to 128 (see "Note" above)

DEBUG queue properties

• Nodes: 4
• Maximum nodes per job: 4 (16 cores)
• Maximum nodes per user: 4
• Maximum wall time: 15 minutes
• Maximum CPU time: 15 minutes per core per job
• Maximum running jobs in queue: 1
• Maximum idle jobs in queue: 1

IEDC queue properties (group restricted access)

• Nodes: 512
• Maximum nodes per job: 256
• Maximum nodes per user: 256
• Maximum wall time: 336 hours (14 days)
• Maximum CPU time: 344064 hours (14 days) per core per job
• Maximum running jobs in queue: Unlimited
• Maximum idle jobs in queue: Up to 128 (see "Note" above)

Important: Default wall time in all queues is two hours; request more if you need more. The default limit on running jobs is 16 per user. Please contact the administrators if you feel you need a higher limit.

Mail usage

UITS does not provide a production mail service on the Big Red cluster; however, LoadLeveler communicates via email. Mail forwarding should be configured during account creation, but if you are receiving mail on the cluster, you may need to set up a .forward file for your account. For help, see How do I forward my mail from a Unix account?

If you plan to run a large number of jobs through Big Red in a relatively short period of time and will be forwarding notifications to an external mail server, make sure the external mail server can handle the load. If you are not absolutely certain that the external email server can handle a large volume of mail, include this line in your jobs:

Scheduled downtime

Big Red maintenance windows will be scheduled for the first Tuesday of each month. A window will be used only if necessary; an announcement regarding the status of maintenance will be posted the preceding Friday morning on both the Big Red Message of the Day and the Scheduled Downtime page.

Questions and comments

The policies described in this document were established with the goal of providing a stable, powerful, and efficient research computing environment for IU faculty, graduate students, and staff. If you have any questions or comments about these policies, UITS welcomes your input. Please contact High Performance Systems.