NCO User Guide - SourceForge

1 NCO User GuideA suite of netCDF operatorsEdition , forNCOV ersion 2022by Charlie ZenderDepartments of Earth System Science and Computer ScienceUniversity of California, IrvineCopyrightc 1995 2022 Charlie is the first edition of theNCO User Guide ,and is consistent with version 2 by Charlie ZenderDepartment of Earth System Science3200 Croul HallUniversity of California, IrvineIrvine, CA 92697-3100 USAP ermission is granted to copy, distribute and/or modify this document under the terms oftheGNUFree Documentation License, Version or any later version published by the FreeSoftware Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-CoverTexts. The license is available online gratefully acknowledge support forNCOdevelopment and maintenance provided bythese institutions and programs:DOE ACME DE-SC0012998,LLNL-B625903,LLNL-B632442,N ASA ACCESS NNX12AF48 AandNNX14AH55A, andNSF SEI IIS-0431203,AGS-1541031,andOAC-2004993.

2 This research was supported as part of the Energy Exascale EarthSystem Model (E3SM) project, formerly known as Accelerated Climate Modeling for Energy(ACME), funded by the Department of Energy, Office of Science, Office of Biologicaland Environmental Research. This material is based upon work supported by the NationalScience original author of this software, Charlie Zender, wants to improve it with the help ofyour suggestions, improvements, bug-reports, and Zender<surname at uci dot edu>(yes, my surname is zender)Department of Earth System Science3200 Croul HallUniversity of California, IrvineIrvine, CA 92697-3100iTable of How to Use This Operating systems compatible CompilingNCOfor Microsoft Symbolic netCDF2/3/4 and HDF4/5 Help Requests and Bug Operator Climate Model Temporary Output Appending Simple Arithmetic and Statistics vs.

3 Averagersnces,ncra, Large Numbers of Large Memory Single and Multi-file Memory Shared Metadata OpenMP Command Line Truncating Long Sanitization of Specifying Input Specifying Output Accessing Remote Retaining Retrieved File Formats and File Determining File File Large File Subsetting Subsetting Coordinate Group Path Deletion, Truncation, and Flattening of Moving Dismembering Checking C and Fortran Index Record Wrapped Auxiliary Grid Options Climatology and Bounds UDUnits Rebasing Time Multiple Record Missing of Current Practices for Real World Lossy Linear Precision-Preserving MD5 Buffer RAM Unbuffered Packed Packing (Default)

4 Unpacking Packing and Unpacking of Packed Data by Other Operation Type Automatic type Promoting Single-precision to Manual type Batch Global Attribute History File List Operator Reference Arithmetic Syntax Left hand Arrays and Value Number if Print & String Missing values Methods and Where Include UDUnits Irregular Bilinear GSL special GSL GSL least-squares GSL GSL random number Examples Intrinsic mathematical Operator precedence and ID makebounds() solarzenithangle Attribute Binary Climatology Reshaping mode, aka Climos and Other , Dedicated Nodes nodeMPI-mode doesncclimodo?..260 Assumptions, Approximations, and Algorithms (AAA) Employed.

5 Ensemble Ensemble File Kitchen specific Filters Permute Dimensions and Unpacking Record Record not regridded specific Weighted Mask Normalization and Proposals for Institutional Quick Daily data in one Monthly data in one One time point one Multiple files with multiple time CMIP5 Combine Global Distribution of Long-term Annual Average over Monthly Add Coordinates CCSM the result of software needs that arose while I worked on projects funded byNCAR,NASA, andARM. Thinking they might prove useful as tools or templates to others, itis my pleasure to provide them freely to the scientific community. Many users (most ofwhom I have never met) have encouraged the development ofNCO. Thanks espcially to JanPolcher, Keith Lindsay, Arlindo da Silva, John Sheldon, and William Weibel for stimulatingsuggestions and correspondence.

6 Your encouragment motivated me to complete theNCOUser Guide . So if you likeNCO, send me a note! I should mention thatNCOis not connectedto or officially endorsed by Unidata,ACD,ASP,CGD, or ZenderMay 1997 Boulder, ColoradoMajor feature improvements entitle me to write another Foreword. In the last five yearsa lot of work has been done to now an open source project and appearsto be much healthier for it. The list of illustrious institutions that do not endorseNCOcontinues to grow, and now ZenderOctober 2000 Irvine, CaliforniaThe most remarkable advances inNCOcapabilities in the last few years are due to con-tributions from the Open Source community. Especially noteworthy are the contributionsof Henry Butowsky and Rorik ZenderJanuary 2003 Irvine, CaliforniaNCOwas generously supported from 2004 2008 by US National Science Founda-tion (NSF) grantIIS-0431203 ( ).

7 This support allowed me to maintain and extend coreNCOcode, User Guideand others to advanceNCOin new directions: Gayathri Venkitachalam helped implementMPI; Harry Mangalam improved regression testing and benchmarking; Daniel Wang de-veloped the server-side capability,SWAMP; and Henry Butowsky, a long-time contributor,developedncap2. This support also ledNCOto debut in professional journals and personal and professional contacts made during this evolution have been ZenderMarch 2008 Grenoble, FranceThe end of theNSF SEIgrant in August, 2008 curtailedNCOdevelopment. Fortunatelywe could justify supporting Henry Butowsky on other research grants until May, 2010 whilehe developed the keyncap2features used in our climate research. And recently theNASAACCESS program commenced funding us to support netCDF4 group functionality. ThusNCOwill grow and evade bit-rot for the foreseeable continue to receive with gratitude the thanks ofNCOusers at nearly every scientificmeeting I attend.

8 People introduce themselves, shake my hand and extolNCO, often effu-sively, while I grin in stupid embarassment. These exchanges lighten me like I daydream how many hoursNCOhas turned from grunt work to productiveresearch for researchers world-wide, or from research into early happy-hours. It s a ZenderApril, 2012 Irvine, CaliforniaTheNASA ACCESS2011 program generously supported (Cooperative AgreementNNX12AF48A)NCOfrom 2012 2014. This allowed us to produce the first iteration ofa Group-oriented Data Analysis and Distribution (GODAD) software ecosystem. Shiftingmore geoscience data analysis toGODADis a long-term plan. Then theNASA ACCESS2013program agreed to support (Cooperative Agreement NNX14AH55A)NCOfrom 2014 support permits us to implement support for Swath-like Data (SLD). Most recently,theDOEhas funded me to implementNCOre-gridding and parallelization in support oftheirACME program.

9 After many years of craftingNCOas an after-hours hobby, I finallyhave the cushion necessary to give it some real attention. And I m looking forward to thisnext, and most intense yet, phase ZenderJune, 2015 Irvine, CaliforniaTheDOEE nergy Exascale Earth System Model (E3SM) project (formerlyACME) hasgenerously supportedNCOdevelopment for the past four years. SupportingNCOfor amission-driven, high-performance climate model development effort has brought unprece-dented challenges and opportunities. After so many years of staid progress, the recentdevelopment speed has been both exhilirating and ZenderMay, 2019 Laguna Beach, CaliforniaTheDOE E3 SMproject has supportedNCOdevelopment and maintenance since is an eternity in the world of research funding! Their reliable support has enabled usto add cutting-edge features including quantization, vertical interpolation, and support formultiple regridding weight-generators.

10 RecentlyNSFsupported us to enable user-friendlysupport for modern compression algorithms that can make geoscience data analysis greenerby reducing dataset size, and thereby storage, power, and associated greenhouse gas emis-sions. I am grateful for this this agency support that inspires me to create new featuresthat help my amazing colleagues pursue their scientific ZenderJuly, 2022 Laguna Beach, California5 SummaryThis manual describesNCO, which stands for netCDF a suite of programsknown asoperators. Each operator is a standalone, command line program executed atthe shell-level like, ,lsormkdir. The operators take netCDF files (includingHDF5files constructed using the netCDFAPI) as input, perform an operation ( , averaging orhyperslabbing), and produce a netCDF file as output. The operators are primarily designedto aid manipulation and analysis of data.