f90nml - A Fortran namelist parser, generator, and editor

Overview

f90nml - A Fortran namelist parser, generator, and editor

A Python module and command line tool for parsing Fortran namelist files

https://travis-ci.org/marshallward/f90nml.svg?branch=master https://ci.appveyor.com/api/projects/status/bcugyoqxiyyvemy8?svg=true https://coveralls.io/repos/marshallward/f90nml/badge.svg?branch=master

Documentation

The complete documentation for f90nml is available from Read The Docs.

http://f90nml.readthedocs.org/en/latest/

About f90nml

f90nml is a Python module and command line tool that provides a simple interface for the reading, writing, and modifying Fortran namelist files.

A namelist file is parsed and converted into an Namelist object, which behaves like a standard Python dict. Values are converted from Fortran data types to equivalent primitive Python types.

The command line tool f90nml can be used to modify individual values inside of a shell environment. It can also be used to convert the data between namelists and other configuration formats. JSON and YAML formats are currently supported.

Quick usage guide

To read a namelist file sample.nml which contains the following namelists:

&config_nml
   input = 'wind.nc'
   steps = 864
   layout = 8, 16
   visc = 1.0e-4
   use_biharmonic = .false.
/

we would use the following script:

import f90nml
nml = f90nml.read('sample.nml')

which would would point nml to the following dict:

nml = {
    'config_nml': {
        'input': 'wind.nc',
        'steps': 864,
        'layout': [8, 16],
        'visc': 0.0001,
        'use_biharmonic': False
    }
}

File objects can also be used as inputs:

with open('sample.nml') as nml_file:
    nml = f90nml.read(nml_file)

To modify one of the values, say steps, and save the output, just manipulate the nml contents and write to disk using the write function:

nml['config_nml']['steps'] = 432
nml.write('new_sample.nml')

Namelists can also be saved to file objects:

with open('target.nml') as nml_file:
   nml.write(nml_file)

To modify a namelist but preserve its comments and formatting, create a namelist patch and apply it to a target file using the patch function:

patch_nml = {'config_nml': {'visc': 1e-6}}
f90nml.patch('sample.nml', patch_nml, 'new_sample.nml')

Command line interface

A command line tool is provided to manipulate namelist files within the shell:

$ f90nml config.nml -g config_nml -v steps=432
&config_nml
   input = 'wind.nc'
   steps = 432
   layout = 8, 16
   visc = 1.0e-4
   use_biharmonic = .false.
/

See the documentation for details.

Installation

f90nml is available on PyPI and can be installed via pip:

$ pip install f90nml

The latest version of f90nml can be installed from source:

$ git clone https://github.com/marshallward/f90nml.git
$ cd f90nml
$ pip install .

Package distribution

f90nml is not distributed through any official packaging tools, but it is available on Arch Linux via the AUR:

$ git clone https://aur.archlinux.org/python-f90nml.git
$ cd python-f90nml
$ makepkg -sri

Volunteers are welcome to submit and maintain f90nml on other distributions.

Local install

Users without install privileges can append the --user flag to pip from the top f90nml directory:

$ pip install --user .

If pip is not available, then setup.py can still be used:

$ python setup.py install --user

When using setup.py locally, some users have reported that --prefix= may need to be appended to the command:

$ python setup.py install --user --prefix=

YAML support

The command line tool offers support for conversion between namelists and YAML formatted output. If PyYAML is already installed, then no other steps are required. To require YAML support, install the yaml extras package:

$ pip install f90nml[yaml]

To install as a user:

$ pip install --user .[yaml]

Contributing to f90nml

Users are welcome to submit bug reports, feature requests, and code contributions to this project through GitHub. More information is available in the Contributing guidelines.

Comments
  • Add file arguments

    Add file arguments

    Some API questions are how to handle force= and the default output name for patching; I decided to ignore force for files. For the latter issue, I raise ValueError when nml_fname is a file since there is no sane default.

    (personally, if it was my library, I would rip out the default name functionality entirely, and make it so that when there is no output file or path, it will still build and return the patched Namelist object without writing a patched namelist file. But that's just me)

    Addresses #25 (although there is no string API here, at least yet)

    opened by ExpHP 25
  • Order of repeated blocks not preserved when mixed with other named blocks

    Order of repeated blocks not preserved when mixed with other named blocks

    File file contains

    &block00
    i=1
    k=2
    &end
    &block01
    m=0
    n=1
    &end
    &block00
    j=3
    z=4
    &end
    

    The statement f90nml.write(f90nml.read(file),outfile) will result in outfile containing

    &block00
    i=1
    k=2
    /
    &block00
    j=3
    z=4
    /
    &block01
    m=0
    n=1
    /
    

    where order of namelist blocks has not been preserved.

    opened by jpf-x 24
  • Strings starting with + are parsed as a list

    Strings starting with + are parsed as a list

    I have a file formatted this way

    &iostuff
    output_folder = +out
    /
    

    When parsed, ['iostuff']['output_folder'] equals ['+', 'out']. I guess this is done so that if the variable out is defined somewhere else, this list is replaced by the corresponding numerical value. But if it isn't defined, the string '+out' should be returned instead. Maybe a simple solution would be having a way to tell f90nml that some variables are strings and should be considered as such.

    opened by amorison 24
  • Namelist with repeated groups doesn't convert to string

    Namelist with repeated groups doesn't convert to string

    Trying to convert a namelist with repeated groups to a string seems to fail. If I do the following:

    >>> import f90nml
    >>> nml = f90nml.reads("""
    ... &foo
    ...  a = 1
    ... /
    ... &foo
    ...  b = 2
    ... /
    ... """)
    >>> str(nml)
    "Namelist([('foo', [Namelist([('a', 1)]), Namelist([('b', 2)])])])"
    

    I was expecting str(nml) to return "&foo\n a = 1\n/\n&foo\n b = 2\n/\n", as is the case when the group names do not repeat. Interestingly, if I use nml.write('out.nml') the the output file is formatted correctly.

    opened by chunderbunny 21
  • Write bug in v0.20

    Write bug in v0.20

    I'm getting an odd bug when trying the write the following namelist:

     &a
     b%c_d_E(1)%ID = 1,
     b%c_d_E(2)%ID = 2,
     /
    

    Code is:

        import f90nml
        data = f90nml.read('test.nml')
        with open('test_RESAVED.nml','w') as f:
            data.write(f, force=True)
    

    Error message is:

    Traceback (most recent call last):
      File "test.py", line 309, in <module>
        data.write(f, force=True)
      File "C:\Anaconda3\lib\site-packages\f90nml\namelist.py", line 232, in write
        self.write_nmlgrp(grp_name, grp_vars, nml_file)
      File "C:\Anaconda3\lib\site-packages\f90nml\namelist.py", line 253, in write_nmlgrp
        for v_str in self.var_strings(v_name, v_val, v_start=v_start):
      File "C:\Anaconda3\lib\site-packages\f90nml\namelist.py", line 302, in var_strings
        v_strs = self.var_strings(v_title, f_vals, v_start=v_start_new)
      File "C:\Anaconda3\lib\site-packages\f90nml\namelist.py", line 313, in var_strings
        i_s = v_start[::-1][len(v_idx)]
    TypeError: 'NoneType' object is not subscriptable
    

    Oddly, when I change c_d_E to c_d_e, it works fine.

    I just updated to the latest release using pip. I've been able to read variables like this before, so something has changed.

    opened by jacobwilliams 20
  • Add a dependency on the yaml module

    Add a dependency on the yaml module

    Trying to use f90nml after installing on macOS via pip3 install f90nml with the -f yaml flag results in:

    f90nml: error: YAML module could not be found.
    

    I think you should just add an explicit dependency on pyyaml. Or, better yet, if there's a way to add a recommended (installed by default) dependency on pyyaml that someone could opt out of. (I'm not familiar enough with python packaging to know for certain if this is possible one way or the other.)

    opened by zbeekman 19
  • Change to accept NumPy arrays and similar objects

    Change to accept NumPy arrays and similar objects

    Hi there! We've been using f90nml in Pyrokinetics to read/write input files to plasma physics codes, and we ran into an issue where NumPy 0D arrays would propagate unnoticed into a Namelist and cause an exception to be thrown on calling Namelist.write. I've added a small change to Namelist._f90repr to detect 0D arrays and convert them to their equivalent primitive type. Nothing changes if NumPy is not installed.

    Let me know if you'd like me to add further test cases, or if you'd prefer to avoid including these sorts of edge cases in your library.

    opened by LiamPattinson 18
  • Optimizing performance

    Optimizing performance

    I was wondering if there were possibilities for optimizing performance for very large namelist files. I'm in the process of doing some benchmarks for some large files (say around 1500 lines, with multiple namelists and pretty much all variable types). I can only get around 1.7 calls of f90nml.read() per second. I assume that it's the parsing and/or creating of the structures that is the bottleneck (actually reading in the lines should only take a fraction of a second), but I'm going to investigate further.

    One thing I was wondering if parsing of different namelists in the same file could take place in parallel? I don't know if such a thing is possible or not, but if it is that might be something to explore. Maybe this is something I can try and contribute to, rather than just reporting bugs and asking for features!

    enhancement 
    opened by jacobwilliams 17
  • repetition of null value bug

    repetition of null value bug

    Hi Marshall, Thanks for this great tool!

    The bug occurs for the following variable input line: AXFCLN = 3.0, 3.0, 48*, The 48* denotes a repetition of 48 null values Upon .write(), it is being turned into:
    AXFCLN = 3.0, 3.0, ','*,

    One clue is that everything is fine (output = input) if the trailing comma is removed prior to processing.

    Processing attached files as follows: nml=f90nml.read('repetition_of_null_win.nml') f90nml.patch('repetition_of_null_win.nml', nml, 'patched_data_win.nml') and likewise for the unix line-endings version.

    I'm using latest master commit 151438f on Oct 31, 2019 (ver 1.1.2) My sys.version is: 3.7.5 (tags/v3.7.5:5c02a39a0b, Oct 15 2019, 00:11:34) [MSC v.1916 64 bit (AMD64)] I'm running Win10/64/Pro.

    patched_data_unix.txt patched_data_win.txt repetition_of_null_win.txt repetition_of_null_unix.txt

    opened by frankeye 15
  • whitespace between indices for array in namelist cannot be parsed by nag

    whitespace between indices for array in namelist cannot be parsed by nag

    When writing namelists containing nested structures that need to be indexed, e.g.

    idx_nml
       v(1, 1) = 5
       v(2, 1) = 5
    /
    

    the resulting namelist file cannot be parsed by a program compiled with nag (tested here with version 6.1) , if there is a whitespace between the indices. I was wondering, if that whitespace can be omitted?

    opened by gitporst 15
  • f90nml.read() crashes for namelists without a terminating EOL.

    f90nml.read() crashes for namelists without a terminating EOL.

    I'm trying to transform a list of configuration files into a list of f90nml.Namelist objects. StopIteration is being raised by Tokenizer.update_chars().

    For example, let's say I have two nml files, foo.nml and bar.nml, this bug can be triggered by :

    [f90nml.read(s) for s in ['foo.nml','bar.nml']]
    
    opened by neutrinoceros 14
  • patching a repeated section on a namelist returns an unclear message

    patching a repeated section on a namelist returns an unclear message

    Related to esm-tools/esm_tools#843

    Problem description

    When loading a namelist with repeated sections, and then I try to patch one or several repeated sections, I get the following error:

    AttributeError: 'Cogroup' object has no attribute 'update'
    

    I understand it is not possible to update repeated sections, as it is not clear which one f90nml should patch. However, I find the error message misleading.

    Example

    To reproduce the error you can write the following fort.4 namelist:

    &NAMDIM
        NPROMA = -8, 
    /
    
    &NAMPAR0
        LSTATS = true,
        LDETAILED_STATS = false,
        LSYNCSTATS = false,
        MP_TYPE = 2,
        MBX_SIZE = 32000000,
        NPROC = 144,      ! Number of MPI ranks (cores of no OpenMP)
        NOUTPUT = 1,
    /
    
    &NAEPHY
        LEPHYS = true,
        LERADI = true,
        LELAIV = false,
    /
    
    &NAERAD
        NRPROMA = -8, 
        CRTABLEDIR = "./rtables/",  ! Modify this for your installation, note trailing /
    /
    
    &NAMPAR0
    /
    &NAEPHY
    /
    &NAERAD
    /
    

    Then you can use the following python3 script to reproduce the error:

    import f90nml
    
    nml = f90nml.read("fort.4")
    
    changes = { 
        "namdim": {
            "nproma": 0
        }   
    }
    
    nml.patch(changes)
    
    print(nml)
    print("Changing variables in non-repeated sections works!")
    print()
    print("Here comes the problem:")
    
    changes = { 
        "nampar0": {
            "lstats": False
        },  
        "naephy": {
            "leradi": False
        },
        "naerad": {
            "nrproma": 0
        },
    }
    
    nml.patch(changes)
    

    Suggested solution

    Ideally, the user should be reported which variables are problematic in this context. I have sketched a very rough solution with the desired behaviour for these lines: https://github.com/marshallward/f90nml/blob/2a8663aa7dfc06a446fbab6aae20d928f694f3fb/f90nml/namelist.py#L649-L658

    The rough solution:

        def patch(self, nml_patch):
            """Update the namelist from another partial or full namelist.
    
            This is different from the intrinsic `update()` method, which replaces
            a namelist section.  Rather, it updates the values within a section.
            """
    +       conflicting_repeated_secs = [] 
            for sec in nml_patch:
                if sec not in self:
                    self[sec] = Namelist()
    -           self[sec].update(nml_patch[sec])
    +           try: 
    +               self[sec].update(nml_patch[sec])
    +           except AttributeError:
    +               conflicting_repeated_secs.append(sec)
    +       if conflicting_repeated_secs:
    +           raise AttributeError(
    +               "The following sections are repeated in the namelist and cannot be "
    +               f"updated: {conflicting_repeated_secs}"
    +           )  
    
    opened by mandresm 1
  • patching of fortran types only works for first occurence

    patching of fortran types only works for first occurence

    Given the namelist file setup.nml

    &setup
     test%var1 = 1
     test%var2 = 2
    /
    

    and the patch

    patch={'setup': {'test':{'var2': 3}}}
    

    the call

    f90nml.patch('setup.nml', patch, 'patched_setup.nml')
    

    does not patch anything.

    However, patching the first occurrence of test% in setup.nml works

    patch={'setup': {'test':{'var1': 3}}}
    
    opened by rainbowsend 1
  • Problem when loading a namefile

    Problem when loading a namefile

    Hi I am trying to load a namefile from the simulation code Ramses (name = namelist.txt). It works most of the time but I have a run when this fails and gives the following message (see below). This may be just that my namefile is wrongly formatted. Is there a way for me to easily read the namelist and see where the formatting may be wrong?

    Apologies for the naive question.

    Thanks for any tips that may help.

    Eric

    ====================================== [within ipython]

    nm = "namelist.txt" f90nml.read(nm)


    AssertionError                            Traceback (most recent call last)
    Input In [10], in <cell line: 1>()
    ----> 1 f90nml.read(nm)
    
    File ~/.local/lib/python3.9/site-packages/f90nml/__init__.py:31, in read(nml_path)
         13 """Parse a Fortran namelist file and return its contents.
         14 
         15 File object usage:
       (...)
         28 >>> nml = parser.read(nml_file)
         29 """
         30 parser = Parser()
    ---> 31 return parser.read(nml_path)
    
    File ~/.local/lib/python3.9/site-packages/f90nml/parser.py:281, in Parser.read(self, nml_fname, nml_patch_in, patch_fname)
        279 nml_file = open(nml_fname, 'r') if nml_is_path else nml_fname
        280 try:
    --> 281     return self._readstream(nml_file, nml_patch)
        282 except StopIteration:
        283     raise ValueError('End-of-file reached before end of namelist.')
    
    File ~/.local/lib/python3.9/site-packages/f90nml/parser.py:388, in Parser._readstream(self, nml_file, nml_patch_in)
        385 # Set the next active variable
        386 if self.token in ('=', '(', '%'):
    --> 388     v_name, v_values = self._parse_variable(
        389         g_vars,
        390         patch_nml=grp_patch
        391     )
        393     if v_name in g_vars:
        394         v_prior_values = g_vars[v_name]
    
    File ~/.local/lib/python3.9/site-packages/f90nml/parser.py:554, in Parser._parse_variable(self, parent, patch_nml)
        551 self._update_tokens()
        552 self._update_tokens()
    --> 554 v_att, v_att_vals = self._parse_variable(
        555     v_parent,
        556     patch_nml=v_patch_nml
        557 )
        559 next_value = Namelist()
        560 next_value[v_att] = v_att_vals
    
    File ~/.local/lib/python3.9/site-packages/f90nml/parser.py:566, in Parser._parse_variable(self, parent, patch_nml)
        561     self._append_value(v_values, next_value, v_idx)
        563 else:
        564     # Construct the variable array
    --> 566     assert self.token == '='
        567     n_vals = None
        569     self._update_tokens()
    
    AssertionError: 
    
    opened by emsellem 3
  • Cannot parse a namelist without the right boundary of the array

    Cannot parse a namelist without the right boundary of the array

    I have a namelist like this ( q is an eight dimensional array, q(8) ) :

    &example
    q(1) = 1, 2, 3
    q(4) = 4, 5, 6
    /
    

    Fortan can parse correctly and get q = 1, 2, 3, 4, 5, 6, but f90nml cannot parse correctly: Namelist([('example', Namelist([('q', [1, None, None, 4])]))])

    opened by coreur 18
  • Parsing error using array assignment in namelist

    Parsing error using array assignment in namelist

    One of the namelists I am trying to parse using f90nml contains an array assignment of the following syntax

    &example
      arr(1:2)%foo =   1.0,   2.0
      arr(1:2)%bar =   3.0,   4.0
    /
    

    If I try to read/parse this namelist

    import f90nml
    f90nml.reads('''
    &example
      arr(1:2)%foo =   1.0,   2.0
      arr(1:2)%bar =   3.0,   4.0
    /
    ''')
    

    I will get the following error

    Traceback (most recent call last):
      File "<pyshell#1>", line 5, in <module>
        ''')
      File "C:\Python\Python37-64\lib\site-packages\f90nml\__init__.py", line 47, in reads
        return parser.reads(nml_string)
      File "C:\Python\Python37-64\lib\site-packages\f90nml\parser.py", line 280, in reads
        return self._readstream(iter(nml_string.splitlines()))
      File "C:\Python\Python37-64\lib\site-packages\f90nml\parser.py", line 359, in _readstream
        patch_nml=grp_patch
      File "C:\Python\Python37-64\lib\site-packages\f90nml\parser.py", line 459, in _parse_variable
        assert v_idx_bounds[0][1] - v_idx_bounds[0][0] == 1
    AssertionError
    
    bug 
    opened by Cory-Kramer 4
  • Assign absolute/relative path to string variable in CLI

    Assign absolute/relative path to string variable in CLI

    It doesn't seem that the CLI supports the use of '/'.

    Using

    f90nml -v DIR='./path/to/dir' -v ODIR='/path/to/dir' input

    will make DIR have the value '.', and ODIR empty. Escaping doesn't work either.

    opened by mateusffreitas 4
Releases(v1.4.1)
Owner
Marshall Ward
Marshall Ward
A command line interface to buy things in stregsystemet

Stregsystemet-CLI This repository is the Stregsystemet CLI, to buy things in Stregsystemet, at AAU. Use of this cli-tool is at your own risk and there

F-klubben 14 Oct 18, 2022
Linux commands Interpreter for Windows and Mac based systems using Python

DBHTermEcIbP Linux commands Interpreter for Windows and Mac based systems using Python Basic Linux commands supported viewing current working director

Vraj Patel 1 Dec 26, 2021
Run an FFmpeg command and see the percentage progress and ETA.

Run an FFmpeg command and see the percentage progress and ETA.

25 Dec 22, 2022
Turdshovel is an interactive CLI tool that allows users to dump objects from .NET memory dumps

Turdshovel Description Turdshovel is an interactive CLI tool that allows users to dump objects from .NET memory dumps without having to fully understa

Leron Gray 41 Jul 27, 2022
🏃 Python3 Solutions of All Problems in GCJ 2022 (In Progress)

GoogleCodeJam 2022 Python3 solutions of Google Code Jam 2022. Solution begins with * means it will get TLE in the largest data set. Total computation

kamyu 12 Dec 20, 2022
A terminal slots programme in PY

PYSlots PyPI and Test PyPI External Links PyPI Test PyPI Install Look directly at the bugs! Version pip install pyslots "Don't look directly at the bu

Luke Batema 4 Nov 30, 2022
a GUI app base on warp-cli for linux

warp cloudflare gui a GUI app base on warp-cli for linux Installation read warp-cli install doc. install warp-cli and register with $ warp-cli registe

Moein Aghamirzaei 58 Jan 01, 2023
Automated CI toolchain to produce precompiled opencv-python, opencv-python-headless, opencv-contrib-python and opencv-contrib-python-headless packages.

OpenCV on Wheels Pre-built CPU-only OpenCV packages for Python. Check the manual build section if you wish to compile the bindings from source to enab

OpenCV 3.2k Jan 04, 2023
CLTools provides various tools and command to use in the terminal.

CLTools provides various tools and command to use in the terminal. As of date, CLTools is only able to generate temporary email addresses and receive emails. There are plans to integrate more tools a

Ashwin Chugh 2 Feb 14, 2022
Python command line tool and python engine to label table fields and fields in data files.

Python command line tool and python engine to label table fields and fields in data files. It could help to find meaningful data in your tables and data files or to find Personal identifable informat

APICrafter 22 Dec 05, 2022
Easy-to-use terminal program that can compile your code.

Description Easy-to-use terminal program that can compile your code. Installition 1. Cloning repository $ git clone https://github.com/DarkJoij/Compil

DarkJoij 1 Oct 21, 2021
🌌 A Python script to generate blog banners from command line.

Auto Blog Banner Generator A Python script to generate blog banners. This script is used at RavSam. The following image is an example of the blog bann

RavSam 10 Sep 20, 2022
A CLI Spigot plugin manager that adheres to Unix conventions and Python best practices.

Spud A cross-platform, Spigot plugin manager that adheres to the Unix philosophy and Python best practices. Some focuses of the project are: Easy and

Tommy Dougiamas 9 Dec 02, 2022
A handy command-line utility for generating and sending iCalendar events

A handy command-line utility for generating and sending iCalendar events This simple command-line utility is designed to generate an iCalendar event,

Baochun Li 17 Nov 21, 2022
A Python package for a basic CLI and GUI user interface

Organizer CLI Organizer CLI is a python command line tool that goes through a given directory and organizes all un-folder bound files into folders by

Caltech Library 12 Mar 25, 2022
gcp-doctor - Diagnostics for Google Cloud Platform

gcp-doctor is a command-line diagnostics tool for GCP customers. It finds and helps to fix common issues in Google Cloud Platform projects. It is used to test projects against a wide range of best-pr

Google Cloud Platform 185 Dec 20, 2022
🔖 Lemnos: A simple, light-weight command-line to-do list manager.

🔖 Lemnos: CLI To-do List Manager This is a simple program that allows one to manage a to-do list via the command-line. Example $ python3 todo.py add

Rohan Sikand 1 Dec 07, 2022
Chat In Terminal - Chat-App in python

Chat In Terminal Hello all. 😉 Sockets and servers are vey important for connection and importantly chatting with others. 😂 😁 I have thought of maki

Shreejan Dolai 5 Nov 17, 2022
Trans is a dependency-free CLI for Google Translate

Trans is a dependency-free CLI for Google Translate

11 Jan 04, 2022
Command line interface for testing internet bandwidth using speedtest.net

speedtest-cli Command line interface for testing internet bandwidth using speedtest.net Versions speedtest-cli works with Python 2.4-3.7 Installation

Matt Martz 12.4k Jan 08, 2023