Miscellaneous: HDF5, YAML, ASDF, pickle (astropy.io.misc
)¶
The astropy.io.misc
module contains miscellaneous input/output routines that
do not fit elsewhere, and are often used by other Astropy sub-packages. For
example, astropy.io.misc.hdf5
contains functions to read/write
Table
objects from/to HDF5 files, but these
should not be imported directly by users. Instead, users can access this
functionality via the Table
class itself (see
Unified file read/write interface). Routines that are intended to be used directly by users are
listed in the astropy.io.misc
section.
astropy.io.misc Package¶
This package contains miscellaneous utility functions for data input/output with astropy.
Functions¶
|
Pickle an object to a specified file. |
|
Unpickle pickled objects from a specified file and return the contents. |
astropy.io.misc.hdf5 Module¶
This package contains functions for reading and writing HDF5 tables that are
not meant to be used directly, but instead are available as readers/writers in
astropy.table
. See Unified file read/write interface for more details.
Functions¶
|
Read a Table object from an HDF5 file |
|
Write a Table object to an HDF5 file |
astropy.io.misc.yaml Module¶
This module contains functions for serializing core astropy objects via the YAML protocol.
It provides functions dump
,
load
, and load_all
which
call the corresponding functions in PyYaml but use the
AstropyDumper
and AstropyLoader
classes to define custom YAML tags for the following astropy classes:
Note
This module requires PyYaml version 3.12 or later.
Example¶
>>> from astropy.io.misc import yaml
>>> import astropy.units as u
>>> from astropy.time import Time
>>> from astropy.coordinates import EarthLocation
>>> t = Time(2457389.0, format='mjd',
... location=EarthLocation(1000, 2000, 3000, unit=u.km))
>>> td = yaml.dump(t)
>>> print(td)
!astropy.time.Time
format: mjd
in_subfmt: '*'
jd1: 4857390.0
jd2: -0.5
location: !astropy.coordinates.earth.EarthLocation
ellipsoid: WGS84
x: !astropy.units.Quantity
unit: &id001 !astropy.units.Unit {unit: km}
value: 1000.0
y: !astropy.units.Quantity
unit: *id001
value: 2000.0
z: !astropy.units.Quantity
unit: *id001
value: 3000.0
out_subfmt: '*'
precision: 3
scale: utc
>>> ty = yaml.load(td)
>>> ty
<Time object: scale='utc' format='mjd' value=2457389.0>
>>> ty.location
<EarthLocation (1000., 2000., 3000.) km>
Functions¶
|
Parse the first YAML document in a stream using the AstropyLoader and produce the corresponding Python object. |
|
Parse the all YAML documents in a stream using the AstropyLoader class and produce the corresponding Python object. |
|
Serialize a Python object into a YAML stream using the AstropyDumper class. |
Classes¶
|
Custom SafeLoader that constructs astropy core objects as well as Python tuple and unicode objects. |
|
Custom SafeDumper that represents astropy core objects as well as Python tuple and unicode objects. |
Class Inheritance Diagram¶
astropy.io.misc.asdf Package¶
The asdf subpackage contains code that is used to serialize astropy types so that they can be represented and stored using the Advanced Scientific Data Format (ASDF).
If both asdf and astropy are installed, no further configuration is required in order to process ASDF files that contain astropy types. The asdf package has been designed to automatically detect the presence of the tags defined by astropy.
For convenience, users can write Table
objects to ASDF files
using the Unified file read/write interface. See Using ASDF With Table I/O below.
Documentation on the ASDF Standard can be found here. Documentation on the ASDF Python module can be found here. Additional details for Astropy developers can be found in Details.
Using ASDF With Table I/O¶
ASDF provides readers and writers for Table
using the
Unified file read/write interface. This makes it easy to read and write ASDF files with
Table
data.
Basic Usage¶
Given a table, it is straightforward to write it out to an ASDF file:
from astropy.table import Table
# Create a simple table
t = Table(dtype=[('a', 'f4'), ('b', 'i4'), ('c', 'S2')])
# Write the table to an ASDF file
t.write('table.asdf')
The I/O registry automatically selects the appropriate writer function to use
based on the .asdf
extension of the output file.
Reading a file generated in this way is equally simple using
read
:
t2 = Table.read('table.asdf')
The I/O registry automatically selects the appropriate reader function based on the extension of the input file.
In the case of both reading and writing, if the file extension is not .asdf
it is possible to explicitly specify the reader/writer function to be used:
t3 = Table.read('table.zxcv', format='asdf')
Advanced Usage¶
The fundamental ASDF data structure is the tree, which is simply a nested
combination of basic data structures (see this
for a more detailed description). At the top level, the tree is a dict
.
The consequence of this is that a Table
object (or any object,
for that matter) can be stored at any arbitrary location within an ASDF tree.
The basic writer use case described above stores the given
Table
at the top of the tree using a default key. The basic
reader case assumes that a Table
is stored in the same place.
However, it may sometimes be useful for users to specify a different top-level
key to be used for storage and retrieval of a Table
from an
ASDF file. For this reason, the ASDF I/O interface provides data_key
as an
optional keyword when writing and reading:
from astropy.table import Table
t = Table(dtype=[('a', 'f4'), ('b', 'i4'), ('c', 'S2')])
# Write the table to an ASDF file using a non-default key
t.write('foo.asdf', data_key='foo')
A Table
stored using a custom data key can be retrieved by
passing the same argument to read
:
foo = Table.read('foo.asdf', data_key='foo')
The data_key
option only applies to Table
objects that are
stored at the top of the ASDF tree. For full generality, users may pass a
callback when writing or reading ASDF files to define precisely where the
Table
object should be placed in the tree. The option for the
write case is make_tree
. The function callback should accept exactly one
argument, which is the Table
object, and should return a
dict
representing the tree to be stored:
def make_custom_tree(table):
# Return a nested tree where the table is stored at the second level
return dict(foo=dict(bar=table))
t = Table(dtype=[('a', 'f4'), ('b', 'i4'), ('c', 'S2')])
# Write the table to an ASDF file using a non-default key
t.write('foobar.asdf', make_tree=make_custom_tree)
Similarly, when reading an ASDF file, the user can pass a custom callback to
locate the table within the ASDF tree. The option in this case is
find_table
. The callback should accept exactly one argument, which is an
dict
representing the ASDF tree, and it should return a
Table
object:
def find_table(tree):
# This returns the Table that was stored by the example above
return tree['foo']['bar']
foo = Table.read('foobar.asdf', find_table=find_table)
Details¶
The asdf subpackage defines classes, referred to as tags, that implement the logic for serialization and deserialization of Astropy types. Users should never need to refer to tag implementations directly. Their presence should be entirely transparent when processing ASDF files.
ASDF makes use of abstract data type definitions called schemas. The tag classes provided here are specific implementations of particular schemas. Some of the tags in Astropy (e.g., those related to transforms) implement schemas that are defined by the ASDF Standard. In other cases, both the tags and schemas are defined within Astropy (e.g., those related to many of the coordinate frames). Documentation of the individual schemas defined by Astropy can be found below in the Schemas section.
Not all Astropy types are currently serializable by ASDF. Attempting to write
unsupported types to an ASDF file will lead to a RepresenterError
. In order
to support new types, new tags and schemas must be created. See Writing ASDF
Extensions for
additional details.