Anticipate 8 KNL nodes in May

We anticipate the arrival of 8 single-node KNL systems in May. Those systems will be made available to all ESP project team members, with no requirement that your home institution has signed the MP-NDA agreement with Intel. Email will go out to everyone when the systems are available, with instructions for access and usage.

Advanced OpenMP training at NERSC Thursday 4 February 2016

NERSC is offering an Advanced OpenMP training day this Thursday. It’s still possible to sign up to attend virtually (Zoom), if you’re interested. If you don’t have a NERSC login id, indicate on the registration form that you heard about the training via the ALCF Early Science Program:

Dear NERSC users,
We are pleased to announce a one-day training on “Advanced OpenMP”:
Speakers: Michael Klemm, Intel (OpenMP Language Committee Member),
    Bronis R. de Supinski, LLNL (Chair of the OpenMP Language Committee)
Time: Thursday, February 4, 9am – 5pm PST
Location: NERSC/LBNL (attend in person or remote)
The OpenMP 4.5 specification was released in November 2015. This tutorial will quickly review the basics of OpenMP programming, with the assumption that attendees understand basic parallelization concepts. Next, the tutorial will detail performance tuning aspects, such as memory placement and affinity considerations, and exploitation of vector units. The presenters will then discuss advanced OpenMP language features in-depth, with an emphasis on tasking, cancellation and new additions in OpenMP 4.5. Using OpenMP from an MPI application and potential pitfalls will also be discussed.
Levels of the tutorial materials are: beginner 10%, intermediate 45%, and advanced 45%. Afternoon hands-on sessions on OpenMP Analysis in VTune and Using SIMD will use the NERSC Cori (Cray XC40 with Intel Haswell processors) and Babbage (Intel Xeon Phi KNC testbed) systems.
Please find more detailed tutorial description, speakers bio, registration, agenda, and in person and remote attendance information at:
In person registration will close on Thursday, January 28.
Best Regards,
Helen He
NERSC User Services

“Theta ESP Wiki” Confluence site is live

The Theta Early Science Program Confluence space is now available. This is meant to function like a Wiki, but using the Confluence technology. Anyone from Theta ESP projects may put content in the space, and for the most part its structure should evolve organically. ALCF will put content here as well, such as slides and videos from ESP meetings, instructions for accessing early hardware when it becomes available, etc. Do not include any NDA information (Intel or Cray) in content you put in this space; we will have another venue for that if/when needed.

The video recordings of the Kick-Off and Training Session 1 videoconferences are available on the Confluence site: Expand “Meetings/Workshops” under the navigation panel at the left and click to bring up the relevant meeting page.

To get access to the site, follow these instructions:

  1. Sign up for a Confluence account:
    • Go to and in the top-right corner of the webpage, click “Sign up”.
    • On the Sign Up screen, enter your contact information, including your preferred email address and click the “Sign up” button.
  2. Obtain access to the Theta ESP Wiki space:
    • Send an email to with your Confluence username requesting access to the Theta ESP wiki.
    • You will be notified via email within two business days when you have been added to the Theta ESP wiki along with the link to the login.
  3. Log into the Theta ESP Wiki space:

Theta ESP Training Session 1 Videoconference

There will be a training-session videoconference on Wednesday 9 September, from 10:00 AM – 5:30 PM CDT.

The purpose of the meeting is to introduce Theta hardware, particularly the Knights Landing CPU, and the software tools to program it. Presentations will be given by Intel and Cray speakers. See agenda on the meeting website.

The intended audience is ALCF Theta ESP project members who will be working on code development and testing for Theta. You will receive an invitation to participate, either directly or through your project’s PI/co-PI. Session content will include Intel NDA material, so attendees’ institutions must have an appropriate nondisclosure agreement signed with Intel.

To register for the videoconference, please visit the Videoconference Registration Site.

Theta ESP Kick-Off Videoconference

There will be a 90-minute Kick-Off videoconference for the Theta ESP projects. Presentations will cover structure of the program, timeline, expectations of the projects, and events.

The videoconference is by invitation only. All ESP project PIs and co-PIs were notified.

NOTE ON BLOG ENTRIES: This is the first entry associated with the Theta ESP. All older entries were for the ESP for what is now our production system, Mira.

Minimum partition size on Mira is 512 nodes; maximum backfill is 8192 nodes

As has been explained in a recent email to Mira users, the minimum partition you can use on the machine is 512 nodes. If you request fewer nodes, you still pay from your allocation for all 512, and the unused nodes are idle. On Cetus, the minimum partition size is 128 nodes.

As some of you exhaust your ESP allocations on Mira, you will notice your jobs going into the “backfill” queue. These are queued with low priority relative to positive-allocation-balance jobs, but will run if resources are available and no normal jobs are available to fit the space. The maximim size job allowed in backfill mode is 8192 nodes.

Cetus Down

There will be an official notice going out soon, but be aware that Cetus is down and will be down for a number of days. This is related to the Vesta downtime—the BG/Q rack that’s currently designated as Cetus is being combined with Vesta to make Vesta a 2-rack system. We have a new rack that will be designated as Cetus. My best estimate is 5 days of downtime for Cetus (yesterday’s notice to vesta-notify and mira-notify lists estimated 5 days downtime for Vesta).

Early Science on Mira is on; time allocations in place

The Early Science period is officially underway. Mira came back online after acceptance testing on the evening of Monday 17 December. After an initial glitch in setting up the computer time allocations for the ESP projects, the correct allocations are now in place. These are what you were awarded as target allocation when your project was selected for the ESP. On Mira, the command

        cbank-list-allocations -u yourUserName -r mira

will show you the amout and usage of your allocation.

Our one-rack test and development machine, Cetus ( is now also available to ESP users.

The Early Science period should last through mid-March. When there is concrete information about the exact transition date, I’ll send out an email with the date and information about how the transition to production usage will impact the Early Science projects. You should have used up your ESP project allocations by then.